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FOREWORD 



The assessment of institutional effectiveness is a challenge to all in- 
stitutions of postsecondary education. Accreditation standards for schools, 
colleges, and universities now require documentary evidence of effective- 
ness in meeting institutional purposes, and public expectations of such 
evidence are implied hy many actions of state legislatures and other agen- 
cies of federal or state government. Postsecondary institutions thus are 
asked to state their purposes in unamhiguous terms and to assess, in 
meaningful ways, the progress made in fulfillment of institutional goals 
and objectives. 

In 1987 the theme of the fifth annual conference on research in 
higher education was carefully chosen with accountability and accredita- 
tion requirements firmly in mind. The conference theme was identified 
simply as, “Assessing Institutional Effectiveness,” and program participants 
were asked to give professional or personal perspectives that would assist 
others in meeting public demands for accountability and/or accreditation. 
The conference, co-sponsored with the Office of Institutional Research and 
Planning, was held at the Georgia Center for Continuing Education. 

Four dinner or luncheon speakers gave personal perspectives on in- 
stitutional effectiveness from their respective positions as: (1) president of 
a rapidly developing and nationally recognized senior college, (2) vice 
president of the nation’s oldest and most successful interstate compact, (3) 
executive vice chancellor of a statewide system of public higher education, 
and (4) associate commissioner for academic affairs and nationally known 
scholar on organizational studies. In her keynote address President Betty 
Seigel gave a commendable overview of Kennesaw College’s emergence as 
a highly effective four-year college “on the perimeter” of a major metropoli- 
tan area. Vice President Bob Stoltz provided a well-informed commentary 
on the regional leadership of the Southern Regional Education Board and 
its efforts to address national demands for improved educational quality. 
Executive Vice Chancellor Dave Spence took a critical, but highly relevant, 
look at state-level efforts to ensure educational quality while increasing 
educational opportunities and access. 

Because her after-dinner presentation went well beyond national 
concerns for assessment. Dr. Ellen Chaffee’s incisive interpretation of 
institutional effectiveness is published as' one of the conference’s invited 



papers. Her paper gives excellent insight into our growing need to inter- 
pret more effectively the purposes and missions of postsecondary institu- 
tions. Her call for the management of meaning should be heard by all 
academic administrators, and her concept of interpretative leadership 
should give presidents, deans, and department heads a mantle they can 
wear with pride. 

In his invited paper Dr. Alton Taylor, professor of higher education 
at the University of Virginia, discusses academic quality in terms that are 
particularly relevant to institutional effectiveness. The concept of quality 
has not been defined concisely, and if we are to assess institutional effec- 
tiveness in attaining academic quality, we must be willing to deal with 
concepts of value and merit. His views complement those of Dr. Chaffee, 
and together their papers give better substance and meaning to the meth- 
ods by which institutional effectiveness can be documented. 

In a special working session on the assessment of student out- 
comes, Dr. Ernie Pascarella, University of Illinois at Chicago, and Dr. Pat 
Terenzini, Institute of Higher Education, gave an excellent analysis and 
interpretation of the methodological issues in assessment and/or measure- 
ment. Institutional effectiveness can indeed be documented best by assess- 
ing changes in student learning and behavior. Dr. Pascarella points out, 
however, that in the assessment of change, our indices (or inferences) can 
be misleading. Dr. Terenzini makes a crucial point that is often missing in 
discussions of assessment: the validity and reliability of assessment meth- 
ods are issues that demand full attention. In brief, institutional research- 
ers, self-study committees, and many others should understand the diffi- 
culties involved in assessing outcomes and they should control or reduce 
such difficulties whenever possible by a more sophisticated application of 
analytic concepts and methods. For institutional researchers, the two pa- 
pers by Pascarella and Terenzini should be required reading. 

Dr. Jim Montgomery, in his unusually practical way, discusses in- 
stitutional effectiveness in terms that all institutional researchers should 
imderstand. Student achievement and effective instruction can be as- 
sessed in many ways, and we are limited only~by our resources and our 
imagination. We should not forget, however, that the purpose of assess- 
ment is the improvement of education. If effective management is needed 
to ensure institutipnal effectiveness, what better way to improve institu- 
tions than helping administrators evaluate and improve themselves? 
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IV 



In an equally practical way, Dr. John Muffo gives an excellent over- 
view of institutional research programs. Institutional research is now con- 
ducted at many different levels of higher education, and information rele- 
vant to institutional analysis is now available from many different sources. 
National agencies and associations are particularly valuable sources of in- 
formation related to self-studies, re-accreditation, and institutional effec- 
tiveness. 

Dr. Richard Howard and Dr. Brenda Rogers, in their respective pa- 
pers, provide exceptional insight into the Southern Association’s new crite- 
ria for accreditation and the response capabilities of institutions seeking 
re-accreditation. As Dr. Howard points out, institutional self-studies now 
include summative evaluation procedures instead of formative evaluation 
efforts. The change in criteria thus calls into play an institution’s ongoing 
institutional research. Institutions already involved in assessment and/or 
evaluation are involved thereby in self-study. In a survey of 311 institu- 
tions scheduled for re-accreditation. Dr. Rogers found significance variance 
in their readiness to conduct a self-study. To an embarrassing extent, in- 
stitutions with existing institutional research offices were not currently en- 
gaged in ongoing assessment or evaluation and had no advantage over 
many institutions without institutional research offices. Dr. Rogers’ study 
(with Dr. Karen (jentemann) should be required reading for all institu- 
tional researchers involved in a self-study. 

Dr. Michael Nettles’ paper addresses institutional effectiveness in 
minority recruitment, performance, and retention. His research implies an 
appreciable imbalance in the institutional distribution of minority stu- 
dents and significant variation in retention and graduation rates. The 
academic performance of minority students is predictable, however, and 
Dr. Nettles’ studies at ETS have identified several significant predictors 
that should encourage institutions of higher education. 

In a panel discussion following Dr. Nettles’ paper. Dr. Ansley Abra- 
ham emphasizes the importance of remedial/developmental programs for 
increasing minority participation. Retention and graduation rates, in 
many institutions, are directly related to the academic support services 
that students receive. Dr. Joe Marks, in his comments, ties minority re- 
cruitment and retention directly to public expectations of institutional 
effectiveness. Efforts to improve institutional effectiveness should not ne- 
gate institutional concern for minority access. Dr. Nathaniel Pugh, a third 
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panelist, questions the representation of minority students in the develop- 
ment of assessment instruments. He reminds us that there are many 
ways of getting through college, and that we should study carefully the 
survival skills of students. 

The publication of invited papers (only) reqmres an explanation 
and an apology to the invited speakers and to six colleagues who con- 
ducted three concurrent workshops during the conference. Dr. Margaret 
Sullivan, associate vice president at (jeorgia State University, conducted a 
workshop for conference participants interested in the institutional effec- 
tiveness of universities. Dr. Ed Rugg, vice president for academic affairs at 
Kennesaw State College, conducted a similar workshop for those inter- 
ested in the self-study of senior colleges. And Dr. Faith Willis, director of 
institutional research at Bnmswick College, organized a workshop on 
assessment in two-year colleges and technical schools. Working with Dr. 
Willis were Dr. Norma Seerley from Gainesville College, Dr. Katherine 
Nisi Zell from Bnmswick College, and Dr. Jerry Shelton from Floyd Col- 
lege. The workshops were well organized, well received, and— according to 
several participants-the most helpful part of the entire conference. 

The reasons for not publishing the speeches and workshop proceed- 
ings are partly technical and partly economic. Preparing camera-ready 
copy on a new computer was a slower process than anticipated, and 
transcribing the luncheon and dinner speeches from audio tape is not a 
process to be highly recommended. Two of the workshops involved exten- 
sive pass-out materials that could not be accommodated, and audience 
participation in the third could not be captured on tape or in print. The 
most understandable reason, however, is budgetary. The lateness with 
which the invited papers have been published is indicative of the funds 
which could be used for printing. Personal apologies, therefore, are very 
much in order! 

In closing, the “quality, merit, or value” of our fifth annual research 
conference should be emphasized. It was, in all respects, an excellent con- 
ference and in publishing the following papers we hope to reach a larger 
and equally appreciative audience. 

Cameron Fincher 
]une 15, 1989 
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MANAGING THE MEANING 
OF INSTITUTIONAL EFFECTIVENESS 



Ellen Earle Chaffee 

Associate Commissioner for Academic Affairs 
North Dakota Board of Higher Education 



The interpretation rather than the assessment of what is going on in 
an institution is the subject of this paper. Such interpretation is as impor- 
tant as assessment and receives far less attention. The purpose of assess- 
ment is to identify areas for improvement. The purpose of interpretation 
is to create a climate in which improvement can occur. With the existence 
of raw data and much more meaningful information, interpretation is the 
next logical level. 

In North Dakota most of the students take the Academic Competency 
Test (ACT) and the average score is 18.5, the national average. Surely 
this is one of the indicators for value-added or competency testing of 
outcomes assessment for K through 12 in North Dakota. Upon examining 
these ACT scores, the investigator finds two different groups of testees: 
those who take high school core curriculum, defined by the ACT to be four 
units of English and three of science, math, and social studies, and those 
who do not. Students who took the core curriculum had an average score 
of 21.5, while those who did not take the core curriculum had an average 
score of 16.5-a five-point spread with more than half the students not 
having taken a core curriculum. Those are the results of an assessment, 
but what do the results mean, what should be done, what is evident about 
K through 12 or higher education? Those are the kinds of interpretative 
questions which proper assessment raises. The next question is where to 
go from here. 

One of the obvious touchstones for effectiveness is the mission of an 
institution. An institution cannot know how well it is doing unless the 
institution understands what it is supposed to be doing. Since this is as 
true for a system of institutions as it is for a single institution, what is the 
mission for a system of institutions such as the North Dakota Board of 
Higher Education? 




10 



2 



Ellen Earle Chaffee 



The North Dakota Board does have a mission statement--among such 
agencies, that kind of statement is not obviously a component of good 
management--and other such agencies have similar statements. The first 
sentence from each of two such statements, North Dakota's and one 
other, illustrate the point that mission statements are not hland, non- 
descript truisms, hut have the power to convey very different messages. 

Here is the first example: “The primary objective of the Board is to 
ensure a diversified, balanced, and high quality system of public higher 
education that provides to citizens who can benefit from an opportunity to 
attend appropriate institutions of higher education.” 

And the second: “The goal of the Board is to afford the citizens of the 
state an efficient, readily accessible, and economical system of high qual- 
ity public higher education through centralized direction and master plan- 
providing for avoidance of unnecessary duplication within the sys- 
tem, for coordination and consolidation.” 

The first focuses on the services, the second on their management. 
The first focuses on effectiveness, the second on efficiency. What does such 
language convey-to Board members about their responsibilities, to insti- 
tutions about their leadership, to the public about its investment? Which 
will elicit more public support? Which will elicit better faculty morale? 
Which will provide for higher-quality higher education? As for the North 
Dakota statement, a major effort to rewrite it is being undertaken. 

Language is important. In fact, every single thing an administrator or 
faculty member does is important in the life of their organization. The 
reason words and deeds are especially important in higher education 
institutions is that the margin is all that is available to work with. The 
organizations’ fundamental purposes are established-organizations can- 
not go into the senior citizen center business to capitalize on changing 
demographics. Their administrators face constraints such as resource 
scarcity, specialization of personnel, tenure, and the professional basis of 
the industry, to name a few. Institutions that tested the limits of such 
constraints during the 1970s often suffered severe consequences in qual- 
ity and credibility. Institutions can only be improved and must be im- 
proved through marginal change. To do that, every word and deed must 
count. 
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Another reason for the importance of language and other tools for 
memaging meaning is that the factors of production are primarily people, 
including custodians, secretaries, faculty, cafeteria workers, and students. 
Some say that higher education is a labor intensive enterprise. A better 
statement is that higher education is an intensely human enterprise. Any 
successful marginal changes are always attributable to the ability to mar- 
shal the energy, talent, and imagination of people; and people are moved 
when they understand and share a dream. 

A brief example illustrates the point. Telecommunications is a major 
(new?) phenomenon in higher education. Many in education have finally 
become convinced that educators should be thinking and doing things 
that capitalize on telecommunications technology, even in North Dakota, 
which has very little technology and no money. North Dakota’s legislative 
study group on the topic had a fairly perfunctory attitude toward its task 
until the head of the public television system got to a key paragraph in 
his testimony. “I am imagining a Thursday afternoon, a few years from 
now,” he said. “A home-bound handicapped woman in Minot is taking an 
economics class at the university in Grand Forks; two bright seniors at 
Bowman high school are learning advanced physics with a class in Fargo; 
a farmer at home in Oakes is receiving a tutorial on computer tools that 
can help him manage his buy-and-sell decisions; managers in the various 
offices of the power compemy are having a seminar with a national expert 
in Minneapolis; . . .” and on he went, painting a vivid word picture of 
what telecommunications could mean in people’s lives. The mood of the 
committee, and the probability that anything worthwhile would come of 
its work, cheuiged dramatically. 

Managing the meeming of institutional effectiveness has three compo- 
nents that this paper will address: the tools available for managing mean- 
ing in institutions, two important concepts of leadership (transforma- 
tional emd interpretive), and common mistakes observed in peoples’ ef- 
forts to interpret mission and manage meaning. 



Tools for Managing Meaning 

Formal mission statements are important and are a vital component 
of the memagement of meaning. But beyond official language, there are 
other tools for managing meaning. 
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Informal Language: Following is a hypothetical story of the importance of 
informal language. Imagine the president of a struggling private liberal 
arts college in a small town on the prairie. At 3 a.m. the fire department 
responds to a call that his college’s gymnasium is burning. By 6 a.m., the 
building is a total loss and the local reporter has called to get the presi- 
dent’s reaction. What does the president say? 

He may be tempted to moan that his capital drive to renovate the 
major classroom building has just begun, winter is coming and the stu- 
dents will be absolutely wild or thoroughly depressed if they have no 
facility for physical activities, and this is a truly tragic day for Harmony 
College. On the other hand, he might say, “Thank heavens it happened at 
a time when no one was in the building—everyone is safe and sound. You 
know, that building was pretty old, and it really wasn’t the kind of place 
that we could share with the local high school and the senior citizens 
centers. It won’t be easy, but we just might be able to work together to 
create something better for all of us.” The point is obvious. 

Feedback and Information: Feedback and information as a category of 
informal language is a major deficiency for most people, for they tend to 
assume that people know why they have made certain decisions. The 
assumption is seldom warranted and can be extremely harmful. 

To illustrate, one university was loaded with tension between the 
faculty and the administration. The faculty had little confidence in the 
president and saw numerous needs unfilled. To take a small but illustra- 
tive example, some of the faculty had offices with no overhead lights and 
no bookshelves, despite repeated appeals. One week, the faculty saw 
trucks and workers galore, all around campus, doing things to the trees. 
Imagine the outrage-there is enough money to tend trees, but not fac- 
ulty. What they did not know, and probably do not know to this day, was 
that the university had not paid for the tree care. The administration 
never explained what was going on or why, and the incident added fuel to 
the faculty’s fire. 

Decisions: It is important not only to explain decisions and actions, but 
also to make the right ones. An example of a president who forgot that 
was the builder-president who had taken a small private college success- 
fully through the growing 1960s and did not quite know how to respond to 
the decline of the early 1970s. One spring he spent $10,000 of college 




13 



Managing The Meaning 



5 



money on the flower bed around the president’s on-campus house. The 
faculty was not pleased when someone calculated that that was more 
than the sum total of all faculty raises that year. 

Behavior: What one does also speaks loudly. A new elementary school 
principal is walking a different child home from school each day so that he 
can get to know them and their families. It is a simple act, obvious in 
intent, not very time-consuming, but unusual enough that although this 
principal is in Wisconsin, news of him was broadcast on the radio in 
North Dakota. 

Physical Facilities: The following are two positive examples of how physi- 
cal facilities of one college sent powerful messages. First, in the midst of 
widespread rumors that the college would soon close, the town also heard 
that the neighboring military school might be acquired by a foreign gov- 
ernment for uncertain purposes. The college president rallied alumni sup- 
port to buy the military school, thereby quelling the rumors of closure, 
making the town very happy indeed, and acquiring badly needed green- 
space and buildings for a brighter future. Second, the renaissance of this 
liberal arts college seemed to require infusion of business programs. The 
president used several means to calm the faculty about the potential loss 
of the liberal arts mission in this change. One such measure was to dedi- 
cate the first renovation on the acquired campus to a fine-arts building. 

A negative example occurred on two campuses where the administra- 
tive officers were physically separated from the main campuses. In one 
case, a mile of empty prairie, owned by the university, was the barrier. In 
the other, the distance was only a couple of blocks, but it was mostly un- 
improved dirt that became a sea of mud every spring and fall. 



Two Concepts of Leadership 

Theoretical leadership literature presents two ideas that clarify how 
and why it is important to manage meaning. The first is transformational 
leadership, an idea first proposed by James McGregor Bums in 1978. 
Transformational leadership involves responding to the higher-order 
needs of individuals in an organization. Birnis asks leaders to recognize 
the merits of Maslow’s hierarchy of human needs as a guide to leadership 
action. The basic needs are for food and shelter. Moving up the scale. 
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people also need a sense of safety, love and respect, and ultimately what 
Maslow calls self-actualization--the expression of their highest principles 
and ahilities. Good leaders, according to Bums, are those who create 
circumstances where individuals can achieve those higher-order needs. 
Such circumstances also make it possible for leaders to learn and satisfy 
their own higher-order needs. Bums’ prescription elicits human leader- 
ship and motivated followers who are themselves, in their own ways, 
leaders. Obviously, it requires a clear focus on the people of an organiza- 
tion, enables them to contribute to the maximum of their abilities, and 
attracts high-quality staff who want to be part of such an enterprise. 

The second idea, interpretive leadership, is not yet well-developed in 
the literature. The term is coined from research on strategic management 
that shows the importance of interpretive strategy for effective organiza- 
tions. Interpretive leadership is similar to transformational leadership in 
its focus on the people of an organization, but perhaps narrower in its 
scope. The purpose of interpretive leadership is to get everyone in the 
organization on the same ship with a clear sense of where the ship is 
headed. It accepts truth in the saying that if you do not know where you 
are going, no wind is favorable. Interpretive leadership responds to the 
frequent calls for a vision of the organization. The sense of direction that 
a vision can bring responds to one of people’s higher-order needs when 
they operate in a collective setting. People need a sense of shared purpose, 
to help them remember why it is worth getting up in the morning to go to 
work, and to provide a context in which they know what kind of work will 
ehcit positive responses for the organization and themselves. In addition 
to shared p\irpose, people need to know what to make of various facts and 
incidents that litter the organizational landscape. The meaning of things 
is not obvious, as in the case of the fire that destroyed the gymnasium. 
Interpretive leadership recognizes what needs to be interpreted and finds 
ways to cast the interpretations that are constmctive and coherent with a 
desired overall pict\ire. 

Interpretive leadership contributes organized direction and a capacity 
to make the kind of marginal changes needed in higher education, and 
every marginal change must be made in the desired direction. Therefore, 
interpretive leadership is the key to effectiveness in organizations. 

Colleges and imiversities need both transformational leadership and 
interpretive leadership-the first to ens\ire high morale among key 
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“factors of production,” the second to ensure coordinated movement in a 
desirable direction. Research has conveyed little about how to carry out 
such leadership; yet the research available shows that both effective lead- 
ership £md effective teaching produce results. 

There are two central tasks of both transformational and interpretive 
leadership. The first central task for leadership is to amswer the question, 
“What direction should we head?” One public college summarized its an- 
swer in the phrase, “to become the number one state college in our state.” 
The goal was often shortened to “becoming number one,” but all involved 
understood that the limiting qualifier, state college in our state, was 
intended. 

This goal demonstrates several of the key characteristics that any 
such answer should have. First, it was consistent with the history and 
tradition of the college. It did not chamge the mission of the institution, 
but rather called forth efforts to achieve the mission to the Nth degree. 
Second, it was responsive to current circumstamces, inside the institution 
and outside it. The college had taken advantage of faculty shortages in 
the 1970s to attract some exceptionally able individuals, yet it was also 
under siege from the difficult economy of the state and fierce competition 
from the numerous other state colleges there. Third, the goal was prag- 
matically feasible. The college had the necessary assets to accomplish it. 

The goal also carried action implications for diverse actors. No matter 
what the job title of any individual, from groundskeeper to business offi- 
cer to faculty, academic deam, £md president, there were ways in which 
each person could help make this the number one state college in the 
state. The goal was open to individual ideas about how to achieve it, 
thereby taking advantage of the maxim that all of us are smarter than 
any of us. 

In addition, the goal elicits pride-it is upbeat, positive, inspirational. 
Recently an academic vice-president had been criticized by some for her 
eternal optimism. After a summer with peers from other institutions at 
the Harvard mamagement institute, she returned persuaded that her col- 
lege’s problems were not nearly as bad as those of others, £md her opti- 
mism was justified. She decided to maintain her attitude but to reject £my 
need to apologize for it. Even in adversity, it is legitimately possible to 
adopt a positive tone. Finally, the goal of this college was succinct and 
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internally consistent. People both inside the college and outside it had an 
image of what a number one state college was and they could see the 
possibilities all around them. 

After deciding where an institution is going, the second key question 
good leaders must answer is, “What kinds of decisions/actions can get us 
there?” The most important factor in answering the question is to recog- 
nize that all decisions and actions are potentially relevant; they must be 
consistent with the vision. If compromises are necessary in the short 
term, they must be explained as compromises, the reasons given, and the 
path to progress defined. Constant communication and feedback are abso- 
lutely necessary. Leaders must remain open to ideas that rise from lower 
levels of the organization. To close that route is to squelch opportunities 
for need-satisfaction among participants in the organization. 



Common Leadership Mistakes 

Although these prescriptions probably sound obvious, they are not so 
easy and many mistakes are made. Following are six common leadership 
mistakes and subtle ways in which those mistakes could have been 
avoided. 

Hidden Agendas, Manipulation for Ulterior Motives: One turnaround 
president came to a small public college from a major research university. 
He came to make a name for himself in a short time so that he could 
return to a major research university in a higher position than he had 
had in such an institution before. His strategy was to try to turn the place 
around as quickly as possible, and his approach was high-energy autoc- 
racy. He worked by organizing projects, writing long memos and direc- 
tives, making non-consultative decisions, and holding high expectations of 
himself and his staff. The place was turning around, but the cost in an- 
tagonism, alienation, and burn-out was becoming high. It was obvious to 
most that the president did not really care about the college, and small 
mutinies were in the making. 

Potential presidents in such situations can take other routes. First, 
they can seriously question whether a presidential opportunity is really 
right for them and their objectives. In this case, the president might have 
done himself and the college a favor by declining the position. Or, a subtle 
alternative to this president’s approach would have been to recognize that 
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fame and higher opportunities come more readily from doing really well 
for an institution than from displaying certain key talents regardless of 
their appropriateness for the situation. 

Trying to be All Things for All People: For example, an urban university 
went all-out to recruit students from other states, despite the fact that the 
luiiversity had no residence halls and that housing in its neighborhood 
was virtually non-existent. Another example is a rural private college 
that started two adult-education centers in other towns, one of them 
dedicated to vocational training to capitalize on the availability of federal 
funds under the CETA program. Its liberal arts image was shattered by 
the strange juxtaposition of unemployed vocationally oriented adults with 
its traditional college students, and it had no expertise to perform its vo- 
cational programs well. The number of liberal arts colleges that were 
hoisted on the petard of rising vocationalism is enormous. 

There are two subtle alternatives to these misguided efforts to do too 
much for too many. The first is doing a sharply limited number of new 
things that express a twist on the mission while attracting new support. 
One private liberal arts college added a business program by capitalizing 
on its strength in economics and ethics, attracting considerable involve- 
ment and support from its corporate trustees. 

The second is to make serious efforts to educate the constituency 
about the value of what an institution already does. Liberal arts colleges 
that waited out the vocational boom with various efforts to find students 
who luiderstood the value of liberal arts and inform others who could 
develop that understanding are now enjo5ring the fruits of national 
reports that convey the same message to a wider audience. 

Refusing to Deal With Incompetence: Nothing imdermines initiative and 
pride more quickly than institutional tolerance of poor performance. In- 
competence is tolerated for many reasons, such as the unpleasantness of 
dealing with it and a false sense that tolerance is a humane alternative to 
firing people. A subtle alternative to these views is recognizing that 
people usually know when they are not doing their jobs well and that 
doing their jobs poorly does not make them happy. Tolerating their in- 
competence is the antithesis of transformational leadership. What is 
needed are humane ways to help people find employment (in or out of the 
institution) that will allow them to shine and grow. 
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Aiming Higher Than You Can Shoot: Sometimes institutions state aspira- 
tions that they cannot reasonably achieve, at least not in the short or 
medium term. That was the case in one university that aimed to he a 
world-class research institution. It was only fifteen years old, was in a 
wealthy state, had an excellent research facvdty in some areas, and in the 
long term may achieve that aim. But the lihreiry was so sparse that one 
faculty member spent over $4,000 annually for books he needed to supple- 
ment the library’s collection, and many of the facvdty were too young to be 
fully competitive for major research grants. As the state entered an eco- 
nomic slump, achievement of the goal receded into a future that few 
expected to see during their cEireers at that institution. The immediate 
effect on the faculty was deep-seated frustration. 

A subtle alternative would have been to narrow the context of the goal 
with a qualif}dng statement-for example, a world-class reseeirch univer- 
sity for the twenty-first century, or an emerging world-class research 
university. Another wovdd have been to admit the shortcomings, set to 
work on fixing them, and report all progress as it occurred. 

Inconsistent Actions: Inconsistencies in higher education Eire rampant. 
Institutions tout the importance of teaching, but promote on the basis of 
research, or recruit adult students without providing evening and week- 
end classes or opening student service offices during that time. No subtle 
alternative exists for this problem. Presidents and institutions must be 
consistent in their multi-faceted efforts to achieve what they say they 
want tp achieve. 

Equating Growth With Size; This fallacy has been well-known at least 
since the publication of E.F. Schumacher’s book Small is Beautiful. Yet 
institutions continue, as Hutchins pointed out in 1936, to degrade them- 
selves in the name of tuition income. The press continues to imply that 
record enrollments are the single indicator of successful institutions. Do- 
nors continue to look at building construction as a major sign of 
effectiveness. What is needed is leadership that takes pride in what exists 
at an institution while inexorably challenging it to become better— not 
bigger. Such leadership is needed in the great majority of colleges and 
universities. 
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INSTITUTIONAL EFFECTIVENESS AND 
ACADEMIC QUALITY 
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Effectiveness and quality are two words that were communicated with 
ease until directed attempts were made to assess them. Currently, few 
scholars have attempted and none has succeeded in a description of these 
terms that will unequivocally suit the needs of institutions, researchers, 
accrediting agencies, state agencies and other external interests. Re- 
search attempts to define and experiment with institutional effectiveness 
have been meager and the use of the construct of quality in academia has 
resulted in individual definitions that suit the context of purpose. Over 
the years, however, some of the most influential thinkers about higher 
education described some clear, usable and useful directions for thinking 
about assessing institutions and academic quality. 

In 1974, Bowen stated: ‘"The idea of accountability is quite simple. It 
means that colleges and universities are responsible for conducting their 
affairs so that the outcomes are worth the cost.” This implies that 
achievements of institutions should be directed toward their missions and 
goals (intended outcomes). Evidence of achievement of these intended 
outcomes should be gathered and their needs, special interests and legal 
duties reported. “The process is bound to be difficult, expensive, and in 
the end subjective and judgmental” (Bowen, 1974). 

Bowen's instructions continue to be helpful to those who have the 
duties for making judgments about how good and effective their academic 
programs and institutions are as a whole and to what degree internal and 
external groups influence decisions for resoiirce acquisitions. The purpose 
of this paper is not to debate the various positions on institutional effec- 
tiveness and academic quality, but is to attempt to create a structiire for 
reflection about effectiveness and academic quiity in higher education. 
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Institutional Effectiveness 

Institutional effectiveness, in the most general sense, is the degree to 
which an institution accomplishes desired effects. In higher education, the 
considered effects of desired accomplishments can be found in a statement 
of goals and objectives. Effectiveness of the institution also must be con- 
sidered in relation to needs and problems of public and special interest 
groups which justify their support and existence. 

Other approaches to defining institutional effectiveness are reported 
by Cameron (1985). These include the Goal Model (including both opera- 
tive and official goals) which defines effectiveness as the extent to which 
an organization accomplishes its goals. This model is supported by Etzi- 
oni, 1961; Campbell, 1977; Price, 1972; and Scott, 1977. Next is the 
Systems Resource Model suggested by Seashore and Yuchtman, 1967. 
This model defines institutional effectiveness as the ability of the institu- 
tion to obtain needed resources. Inputs replace outputs as the primary 
consideration. The third model described by Cameron (1985) is the Proc- 
ess Model suggested by Argyris, 1964; Bennis, 1966; and Likert, 1967. In 
this model effectiveness is defined in terms of how well the internal or- 
ganization operates with its processes and procedures. 

Effectiveness of institutions indicates that institutions as entities have 
effects on internal constituents and the external environment. Effects on 
internal groups include those employed and enrolled. Effects on external 
groups include everybody else. Effects also may be considered in terms of 
physical phenomena (i.e., esthetics of buildings). In a pragmatic sense of 
judging a college or university, the fundamental features of an institution 
are reflected in students (admissions, enrollment, graduation, drop-outs, 
financial aid, housing, social organizations, etc.), faculty (employment, 
tenure, productivity, salaries, benefits, working conditions), facilities 
(laboratories, offices, libraries, etc.) and all the array of support units 
from security to the print shop. Judging effects at the institutional level 
must be comprehensive and insightful. Evidence collected about these 
effects must be clear, understandable, and articulated within a decision 
process clearly understood by those requesting the evidence and those in 
the internal environment. A simplified category for most institutions in- 
cludes the traditional missions of higher education— teaching, research 
and service. Fundamental to public institutions is the teaching mission. 
Ideas regarding the effects of teaching and academic quality, however. 
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are not satisfactory to convince many who are interested in clear, un- 
equivocal evidence to make informed decisions to improve the teaching 
and learning that presently goes on in the college classroom. 

Academic quality is a subjective, relative construct and means how 
good the academic programs are in relation to some agreed-upon standard 
or individual intrinsic value referent within a situational context. 

Traditional approaches used to define and measure academic quality 
include nihilist, reputational, resources, outcomes and value-added (Astin, 
1982). The nihilist approach contends that quality cannot be defined and 
measured. The reputational approach is based on consensus of opinion 
£md is measured in terms of student selectivity, enrollment size, and size 
of graduate faculty. The resources approach is defined and measured in 
terms of highly trained and prestigious faculty, institutional affluence, 
and bright students. The outcomes approach is based on the quality of 
degree recipients in terms of emplo 3 rment, earnings, and important posi- 
tions obtained. Value-added is Emother approach to defining quality and 
resides in the institution’s ability to affect its students favorably in their 
intellectual and personal development. While each of these approaches 
may have relevance to defining and measuring academic quality, there is 
no agreement that these approaches are convincing to most persons need- 
ing to understEmd £md judge academic quality. 

The word “quality” is similar to the word “effectiveness.” Neither 
word has a clear, concise £md unequivocal definition suitable to those who 
search for understanding and those who are charged to assess institu- 
tional effectiveness £md academic quality. There are a few selected fea- 
tures about academic quality, however, which shed some light on the 
dilemma of a lack of consensual understanding. 

An appreciation for the difficulties in the failing to obtain a consensus 
about academic quality may very well rest in the philosophical nature of 
meEmings Emd interpretation. Aristotle’s description of quality has some 
connotations for academic quality. The thought he suggested which may 
be applied to academic programs is that of an affective quality. This 
meEms that academic qualities are capable of producing an “affection in 
the way of perception” (Aristotle, p. 24). It assumes that the perceiver is 
attempting to perceive the qualities of an academic program and not just 
random occurrences. 
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Academic quality is a genuine property in the sense that the quality of 
a progTEim does not depend on whether a person is actually aware of such 
quality. It is real and exists; it is not counterfeit. Primary qualities of a 
program are those which a program can have on its own, independent of 
any perception. Secondary qualities are not in the object but are those 
subjected by the perceiver and may be listed in terms of assumptions or 
intended outcomes. 

While quality has no structure, academic quality is a public expression 
in the sense that judgments of quality are based on certain standards 
which are public stemdards. Perceptions of academic quality cannot go 
unnoticed by either overt behaviors or feelings. Identification of institu- 
tional and progTEim quality is individually sensed by meems of public 
schemata which were learned. 

Academic quality is a correlative of good; so is institutional effective- 
ness. The process of determining academic quality aims, not only at clari- 
f5ring the purposes of the program, but also, if intentions are right, at 
achieving or maximizing values. If a progrEim has high quality or is very 
good, it only mean, that and nothing else. 

The quality of academics rests upon the functionality of the progrEims. 
Because academic quality is always relative to some purpose, and good- 
ness may not be, a program is of high quality if it does a good job of 
fulfilling some purpose or performing some function. No matter how good 
something is, we always want more. 

Institutional effectiveness and academic quality are two terms which 
may easily be confused. While it is natural for academic minds not to 
deliberate or ponder ideas about effectiveness and quality, the pragmatic 
emd political minds are directing institutions to show evidence and con- 
vince the public that their support is highly justified. A scrutiny of merit 
and value might be helpful to reflective thought that explores effective- 
ness and quality in higher education. 



Merit and Value 

Describing and judging merit and worth within an institutional con- 
text is important in completing the process of assessing institutional effec- 
tiveness and academic quality. Merit of institutional effectiveness and 
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academic quality is a judgment of values on their own intrinsic, context- 
free value. Judgments of the worth of effectiveness and quality have 
value within some context of use, application, extrinsic or context-deter- 
mined value. 

While merit judgment is based on intrinsic features of an object, these 
judgments remain relatively stable over time. Worth judgments of effec- 
tiveness and quality depend on internal and external environments which 
change and range over time. Emphasis on institutional effectiveness and 
academic quality for enhancement and progress are constrained by the 
pluralistic values of the internal and external groups. There is usually a 
lack of total agreement, both internal and external, as to judgments of 
effectiveness and the academic quality of a college or university. Merit 
judgments are conducted by a group of experts who agree as to the intrin- 
sic value of academic quality; worth, on the other hand, is judged by 
comparing the institution’s and program’s effect of impact and outcomes 
to some external requirements or standards. 

There are several criteria for determining the worth of institutional ef- 
fectiveness and academic quality. First, there are alleged values which 
are found in public announcements and mission statements. Actual val- 
ues are those deduced from what an institution does, that is, as seen by 
the emphasis and resources allocated to the teaching, research or public 
service missions. Interest values reveal the benefits for the success and 
progress of the institutions. Effectiveness and quality judgments are ex- 
pected to be related to support and resources obtained. Ideal values in- 
clude those virtues of social morality and intentions of good for society. 
These values are intrinsic and difficult to identify for the public. 



Discussion 

Institutional effectiveness is considered at the highest level of thought 
for the organization. Responding to questions for judgments concerning 
what affects an institution is conditioned upon the context of purpose, 
motives and needs for responses by the inquisitor, and expectations of the 
persons in charge of providing evidence of effectiveness. Purposes and 
expected achievements are fundamental to judgments about institutions. 
The most essential precondition of purposes for most, or nearly all, col- 
leges and universities rests upon the academic quality of the institution. 
Academic quality is reflected in complicated, interactive operations which 
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include the students, faculty, facilities, resources and the internal atmos- 
phere within which people work together for the overall progress and 
existence of the institution. 

As complicated and contradictory as the approaches to defining and 
judging effectiveness and academic quality of an institution may appear, 
there are two features for judgment that are fundamental in order for 
judgments to he made; these are purposes and expected achievements. In 
considering purposes and expected achievements, there are those that 
exist because of their intrinsic good and those that have worth values, 
hoth of which are accepted and supported internally and externally hy the 
institution. When conducting assessments of institutional effectiveness, 
judgments of worth are easier to consider and collect evidence to persuade 
decision makers of the value of the institution. Decisions of merit are 
difficult to use in persuading and obtaining consensus because of the 
diversity and pluralistic values and motives of special interest in the 
external environment. 

The academic mindset uses positions of academic merit to convince ex- 
ternal groups of the worth of the institutional welfare and contributions 
to society. Conversely, the public makes decisions about these contribu- 
tions and the role of higher education on bases of worth values. Contexts— 
as to referent needs and problems, timing of social and economic prob- 
lems, trends and individual experiences, and public motives— have a more 
profound influence on worth judgments than those of merit. Attempts to 
render decisions and obtain consensus about institutional effectiveness 
and academic quality based on merit premises are not as easy as those 
attempts to convince judges based on worth values. 

Merit values are usually associated with academic mindsets, but also 
are more prominent in the nonprofessional faculty groups. Professional 
disciplines, by their nature, role, purpose, and expected achievements, can 
make convincing reports more aligned with worth values than nonprofes- 
sional faculties can. 

In closing, academic quality is fundamental to institutional effective- 
ness. The effects of an institution should be judged in context with pur- 
poses for making the judgments, goals and expected achievements and 
the traditional aspects of organizational mission, structure, strategy, and 
governance. Merit and worth values of effects and academic quality 
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should be considered in relation to the duties and motives of intended 
judgments and those making judgments. 

While one may believe that what sustains and enhances higher educa- 
tion are those few influential persons who can convey institutional effec- 
tiveness and academic quality without sa 3 dng a word, the rest of us need 
evidence to persuade us of the merit and worth of colleges and universities. 
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One of the watchwords of higher education in the late 1970s and on 
into the 1980s has heen “accountability.” Public institutions, in particu- 
lar, are being called upon to document their effects in terms of student 
learning and development (Fincher, 1986; Hartle, 1986). Often the impe- 
tus for this call to accountability is state government and the higher 
education coordinating or monitoring boards which, in a number of cases, 
report directly to the governor or the state legislature. In my own state, 
the Illinois Board of Higher Education has studied the condition of under- 
graduate education and made recommendations to the state legislature. 
Two pertinent recommendations are as follows: 

1. Colleges and universities shall conduct regular reviews of under- 
graduate education with emphasis on general education and the 
development of baccalaureate-level skills. 

2. Each college and university shall assess individual student prog- 
ress in meeting the objectives of general education and the devel- 
opment of baccalaureate-level skills, and shall incorporate the re- 
sults of assessment in the reviews of these areas. It is expected 
that colleges and universities will assess student progress at ap- 
propriate intervals and assure that assessment programs reinforce 
the maintenance of academic standards. 

State-mandated student assessment in Illinois is certainly not an iso- 
lated case, as an increasing number of states are moving in a similar 
direction. Moreover, it is likely, based on the widespread impact of the 
recent National Institute of Education Report, Involvement In Learning 
(1984), that the current emphasis on estimating the student outcomes of 
higher education will be with us for the forseeable future. 



'Sections of this paper were previously presented at the 1986 Invita- 
tional Conference of the Educational Testing Service, New York. 



20 



Ernest T. Pascarella 



One of the more recognizable approaches to the assessment of student 
outcomes is the concept of “value-added.” Value-added typically examines 
actual or inferred changes in students’ performance over time. Students 
are assessed for entering competencies on some set of reliable and valid 
instruments, and then are reassessed following a specified period of time 
(e.g., freshman to senior year) or the completion of certain courses, pro- 
grams of study or other educational experiences (McMillan, 1986). Value- 
added has another implication, however, which goes beyond simply look- 
ing at pre-post changes on some measure of interest. Specifically, it at- 
tempts to separate that portion of student growth or development experi- 
ences from that attributable to confounding causes such as differential 
ability or simple maturation. 

In short, value-added entails an estimation of the “net effects” of col- 
lege. It must be acknowledged that its operational definition is likely to 
vary with who uses the term, in what context and for what purpose 
(Ewell, personal communications. May 9, 1986). For some institutions, 
value-added may be inferred from simple freshman to senior changes in 
measures of cognitive development or learning (Lenning, Munday and 
Maxey, 1969; Dumont and Troelstrup, 1981). For other institutions, the 
value of the education they provide may be evidenced largely by the 
accomplishments and retrospective evaluations of graduates (Spaeth and 
Greeley, 1970; Pace, 1974). Still other institutions may see value-added 
largely as I have defined it, and may concentrate on estimating the new 
effects of the collegiate experience (Mentkowski and Strait, 1983). Fur- 
thermore, even within the same institution, it can mean different things to 
different administrative, facility, student and alumni constituencies. Who 
decides what is valuable, for whom it is valuable, and to what it is added? 

Clearly value-added may not mean the same thing to all who use it. 
Indeed, as Ewell has pointed out, the operational definition of the term is 
still quite flexible and vague, and this vagueness may vary in direct 
proportion to its increasing use in public dialogue. 



A Recent Debate on the Value of 
Value-Added Assessment 

Perhaps in part because of the lack of a clear operational [definition of 
the term, the concept of value-added has recently become the focal point 
of a spirited controversy. Some of the most respected names in the 
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postsecondary scholarly community have chosen sides on the issue. Jon- 
athan Warren (1984) has published an enthusiastic and cogent criticism 
of value-added. The gist of his argument is that, while in the abstract the 
logic of value-added has great appeal, in practice it seldom leads any- 
where. The results of value-added analyses, he argues, “are often trivial, 
difficult to make sense of and peripheral to most instructional purposes” 
(p. 10). Warren’s argument focuses largely on one level of an^ysis, the 
use of the value-added approach in assessing course-level instructional 
outcomes. Here he makes some telling points about the questionable prac- 
tice of using pre-post differences as a measure of student learning in 
courses such as upper-division electromagnetic theory, where students 
can be assumed to have little pre-course content knowledge. Simply find- 
ing that students understand course concepts on the final examination is, 
he maintains, sufficient evidence to infer that most of the observed learn- 
ing was a consequence of the course. A similar argument has been made 
by Pace (1985). 

Warren’s recommendation is that we need to abandon the unworkable 
concept of value-added and get on with alternative ways of assessing the 
effects of postsecondary education. Cameron Fincher (1985) is less con- 
vinced that value-added is a blind alley which needs to be abandoned, yet 
he is similarly skeptical in elucidating problems in its implementation. 
These problems, he argues, center on: (1) the development of reliable and 
valid instnunents to measure various educational outcomes; (2) psychom- 
etric problems in assessing change; and (3) conceptual problems in inter- 
preting college effects on achievement when most of the variation on 
achievement may be due to student aptitude, prior achievement and the 
quality of student learning effort rather than to instructional variables. 
While not necessarily ready to abandon the concept, Fincher is nonethe- 
less skeptical about the ability of educators to apply value-added concepts 
to educational issues. He suggests that “if educators could agree on the 
assessable outcomes of higher education, take the time and effort to de- 
velop suitable forms of measurement and assessment, and restructure in- 
structional efforts in terms of explicit instructional objectives, value- 
added concepts of education might then be a solution to some educational 
problems” (p. 398). 

Fincher’s points are well taken. One must wonder, however, whether 
the issues which he elucidates are specific problems of a value-added 
approach to student assessment. A reasonable argument could be made 
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that educational and behavioral research in general have traditionally 
been confronted with these and similar assessment issues. The knotty 
problems of instrument validity and student learning attributes to spe- 
cific instructional practices are longstanding, if not adequately resolved, 
concerns of those interested in the effects of schooling at all levels 
(Wittrock, 1986). 

Responding directly to Warren (1984), Astin (1985) and Ewell (1985) 
have defended value-added as an important contribution to thinking 
about assessing the impact of postsecondary education. Both authors 
readily admit that the approach is not without its problems. At the same 
time, however, they argue that it has conceptual strengths which out- 
weigh these problems. These include: (1) a focus on actual student devel- 
opment rather than typical measures of institutional “prestige” or “qual- 
ity” (e.g., student body selectivity, resources per student, library size); (2) 
the requirement of clearly defining, in conceptual and operational terms, 
the desired outcomes of college or other educational experiences; and (3) 
systematic attempts at assessing the impact of educational experiences. 
They also argue, quite convincingly, that value-added as an approach to 
evaluating the impacts of college has been implemented in a systematic 
manner in only a few postsecondary institutions (e.g., Alverno College). 
Thus, abandoning value-added now wotdd be the equivalent of dismissing 
an idea with a considerably rational appearance “before it has been more 
extensively tried, evaluated and is better understood” (Astin, p. 11). 
Clearly, Astin and Ewell want to avoid throwing the baby out with the 
bathwater— at least until we have a chance to see how the baby matures. 

Astin and Ewell make another point in their defense of value-added. 
Warren’s critique, they suggest, is based almost exclusively on a single 
level of analysis, (i.e., student learning in a single course). Ewell concedes 
the point that there may be course situations involving a well-defined, 
specialized body of knowledge to which students were not previously ex- 
posed where pre-testing makes little or no sense. He further asserts that 
this is far from a valid critique of the entire value-added concept, how- 
ever, since value-added is clearly usefiil in assessing cognitive and devel- 
opmental outcomes of curriciila and educational experiences more broadly 
conceived than individual courses. Indeed, it is in assessing these more 
broad-based effects that value-added has had its most typical application 
in postsecondary education. 
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It seems unlikely that the current debate concerning the utility of 
value-added assessment is over. The term itself is too value-laden and 
perhaps a bit overly pretentious in its public use by educators. In this 
sense it may appear to claim more than many current applications of 
value-added assessment have been able to deliver. As a result it will 
probably continue to invite spirited criticism which, in turn, adds fuel to 
an ongoing debate. If this debate is to contribute light as well as heat, 
however, it may be advantageous to do two things. First, consider redefin- 
ing value-added as the very fundamental and traditional research ques- 
tion: what are the student developmental outcomes associated with expo- 
sure to an educational experience which can be reasonably attributed to 
the educational experience itself and not to other factors? This is the “net 
effects” question, and it is far from being a new concern for educational 
researchers and evaluators (Feldman and Newcomb, 1969; Solmon and 
Taubman, 1973; Bowen, 1977; Astin, 1977). Indeed, a basic issue in edu- 
cational research and evaluation has long been the extent to which stu- 
dent development can be attributed to purposeful educational experi- 
ences. This is also, I believe, the core of what a value-added approach to 
student assessment is about. 

Second, place the discourse about value-added on a different level. If 
there is willingness to accept value-added as a potentially important ap- 
proach to the assessment of student outcomes, then it behooves to con- 
sider ways in which the methodology of the approach might be enhanced 
and sharpened. The remainder of this paper will suggest and discuss a 
number of such methodological enhancements. 



Pre- to Post-Changes; Improvement Can Be Misleading 

It is often the case that the value-added or net effects of an educa- 
tional experience will be inferred from pre- to post-changes (say from 
freshman to senior year) on some accepted measure of student develop- 
ment (e.g., critical thinking, moral reasoning, reflective judgment, ACT- 
COMP scores). Unfortunately, even assuming the reasonable reliability of 
change scores, such mean changes reflect not only the effects of college, 
but also the effects of simple maturation. (Other variables such as his- 
tory, instrument decay, and possibly even regression artifacts if the group 
is extremely low to begin with could also confound interpretation, but it is 
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likely that maturation over time would be the major confounding vari- 
able.) This, of course, means that longitudinal freshman-to-senior changes 
probably overestimate the effect due to college alone (i.e., the unique 
effects of college). 

One possible way to deal with maturation is through the use of a 
control group of subjects followed over the same period of time, but not 
exposed to the particular educational experience. In the situation where 
one is assessing institutional effects, however, reasonably comparable 
control groups not attending college are particularly difficult, if not impos- 
sible, to obtain. For such situations there are alternative cross-sectional 
or combined cross-sectional and longitudinal designs which provide rea- 
sonable controls for maturation. Consider the cross-sectional design 
where freshmen are compared with seniors on a measure of critical-think- 
ing ability. The freshmen, who have not been exposed to the institution, 
would act as a control group for the seniors, who have theoretically bene- 
fited from four years of exposure to it. (To better reflect the entire college 
experience, the measure of critical thinking might be given to freshmen 
upon enrollment in the institution, and to seniors in the final semester or 
quarter of their senior year.) The difference between the average fresh- 
man and average senior scores, statistically adjusted for differences in age, 
could be used to estimate the impact of the institution on critical thinking. 

There are, of course, potential problems with this design which must 
be addressed. First, seniors probably represent a more selective group in 
terms of academic ability since a portion of the least academically profi- 
cient are likely to have flunked-out or to have left for academic reasons. 
Second, there is the possibility of differential recruitment or admission 
criteria being used for the seniors versus the current freshmen (e.g., if the 
institution used a more stringent set of admissions criteria for the current 
seniors as versus the future freshmen, the former might be a more aca- 
demically select group than the latter). This might also lead to systematic 
group biases on a factor such as academic aptitude which, in turn, is 
likely to influence level of critical thinking. While acknowledging that 
there is no ideal way to adjust for such preexisting differences (Lord, 
1967), one might nevertheless select the freshmen to be compared from 
SAT or ACT ranges similar to those of persisting seniors, and accompany 
this with regression analysis to provide aptitude as well as age-adjusted 
critical-thinking means for the freshman and senior groups. The difference 
between the aptitude and age-adjusted means would likely yield a better. 
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though still imperfect, estimate of net institutional effects, than that 
yielded by simply adjusting for age alone. 

Cross-sectional designs such as this have been employed in various 
attempts to separate college effects from those of maturation. It would 
also seem reasonable that such cross-sectional designs could be used in 
conjunction with pre-post longitudinal designs to provide a clearer picture 
of the influence of college versus the influence of maturation. Because the 
simple longitudinal results also include the possibility of maturational 
influences, they might be thought of as the upper-bounds or liberal esti- 
mate of the effect of college. Conversely, the adjusted cross-sectional re- 
sults would tend to statistically remove any joint impacts of the college 
experience and normal student maturation; thus, providing a conserva- 
tive or lower-bounds estimate of the effect of college. The difference be- 
tween the longitudinal and adjusted cross-sectional results might be used 
as an estimate of normal maturation during college. (In the absence of 
longitudinal data one might use the adjusted differences between cross- 
sectional freshman and senior cohorts to represent the upper-bounds esti- 
mate of college effects, the age- and aptitude-adjusted differences to rep- 
resent the lower-bounds estimate, and the difference between the unad- 
justed and adjusted results to represent normal maturation.) 

Another cross-sectional design which has been used to disaggregate 
college effects from age or maturation effects is one which takes advantage 
of the increasing numbers of older, nontraditional-age students in Ameri- 
can postsecondary institutions. In this design, traditional-age freshmen 
(age 18), nontraditional-age freshmen (age 22), traditional -age seniors 
(age 22) and nontraditional-age seniors (age 26) might all be admini- 
stered the measure of critical thinking. The effects of college versus matu- 
ration would be estimated by comparing the freshman to senior differ- 
ences for both traditional- and nontraditional-age students, with age dif- 
ferences between traditional- and nontraditional-age freshmen between 
traditional- and nontraditional-age seniors. Examples of this design have 
been used in estimating the effects of formal education on the develop- 
ment of reflective judgment by Strange (1978). 

These alternative designs are not without flaws. Indeed, there are 
internal validity issues (i.e., what is the real cause of the effects observed) 
associated with both of them. As alternatives to simple pre- and post- 
assessments, however, they do provide somewhat greater control over the 
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confounding influence of student maturation during college. Conse- 
quently, they probably provide more internally valid estimates of the 
value-added or net effects of educational experiences than do simple pre- 
to post-changes. In terms of contributing to the understanding of the 
value-added or net effects of educational experiences, pre- to post-changes 
in the absence of a control group are quite limited. One might conceive of 
them as a necessary but not sufficient condition for docmnenting educa- 
tional impacts. In most instances, the presence of a net educational im- 
pact will be accompanied by positive pre- to post-changes on measures of 
interest. One must be a bit cautious in this regard, however. Recent 
evidence reported by Wolfle (1983), for example, has suggested that the 
general effect of college attendance on mathematics achievement is to 
maintain pre-college levels of competence. Net of other factors, those not 
attending college tend to decline in mathematic competence. Thus a col- 
lege or educational impact may not always be accompanied by pre- to 
post- improvement, or even by pre- to post-change. Indeed, simply looking 
at pre- to post-improvement may, under certain circmnstances, underesti- 
mate the true impact of college. 

It is precisely in this type of situation that the term value-added can 
be misleading or even dysfunctional. As an estimate of college impact, 
value-added implies that something is added to the individual’s level of 
development. In some areas of development, however, the impact of col- 
lege (or other educational experiences) may be to prevent or retard decline 
rather than to induce major positive changes. Consequently approaches to 
value-added assessment which focus only on pre- to post-changes may be 
overlooking important college effects. 

Direct and Indirect Impacts 

It may be argued that value-added assessment is made increasingly 
effective and useful in terms of its policy implications as it becomes more 
specific and focused. At the individual college level, this suggests the 
importance of identif 5 nng specific institutional experiences which enhance 
student development. Assessments for this purpose are often more fine- 
grained in conceptualization and analysis than those which are limited to 
determining the net effect of college or some particular educational expe- 
rience. At the multi-institutional level, we have some exemplary applica- 
tions of this approach in the work of Astin (1968, 1977, 1982; Astin and 
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Panos, 1969) and Pace (1985). Generally this approach involves estimating 
the associations between various measures of the college experience (e.g., 
academic major, academic achievement, extracurricular involvement, in- 
teraction with faculty) and certain outcomes (e.g., standardized measures 
of achievement, graduate degree attainment, self-concept, values) with 
the confoimding influence of student pre-college traits (e.g., aptitude, sec- 
ondary school achievement, social origins) removed statistically. The re- 
sult is an estimate of the effect of a particular collegiate experience inde- 
pendent of differences in student pre-college traits. 

When multiple regression is employed to assess these partial associa- 
tions between college experiences and college outcomes, the resultant re- 
gression coefficients can be interpreted as estimates of the net or direct 
influence of a particular variable (Wolfle, 1985). Thus, in addition to 
estimating the extent to which an institution facilitates the development 
of critical-thinking ability from freshman to senior year, a. value-added 
approach might also attempt to identify those particular institutional 
experiences which have nontrivial net associations with critical thinking. 
The results of these and similar analyses can provide useful information 
in terms of focusing attention on those potentially manipulable collegiate 
experiences which may casually influence the development of critical 
thinking (e.g., particular courses or course-taking patterns, interaction 
with faculty). It is important to stress the “may” in the previous sentence 
since attributions of causality with correlational data are tenuous at best. 

A related issue in assessment is the notion of indirect effects or influ- 
ences. This comes imder the general idea that you are probably doing 
more good than you think you are. It is often the case that specific college 
experiences, or even college itself, is shown to have little independent, 
direct influence on desired outcomes. However, it may be premature and 
misleading to conclude from such evidence that there is no impact due to 
college. The impact may be indirect rather than direct. This can be dem- 
onstrated from some of our own recent research. In a study conducted 
with Pat Terenzini and Lee Wolfle (Pascarella, Terenzini, & Wolfle, 
1986), we considered the influence of a summer orientation program on 
freshman-year persistence. Our hypothesis was that such a program 
might enhance persistence indirectly rather than directly. We expected 
those who attended orientation to have higher levels of early social integra- 
tion during the freshman year, which in turn we anticipated might posi- 
tively influence commitment to the institution and, eventually, persistence. 
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That is generally what happened. When we controlled for background 
differences between those who attended and those who did not attend ori- 
entation, attendance versus nonattendance was found not to have a sig- 
nificant, direct influence on persistence. However, attending orientation 
did have a significant, positive indirect effect on persistence which was 
over three times as large as the direct effect. The indirect influence of ori- 
entation manifests itself through its positive influence on student social 
integration and institutional commitment, which in turn, positively influ- 
enced freshman-year persistence. Had we ignored these indirect effects 
we would have erroneously underestimated the value of orientation. 

Such evidence suggests that certain institutional impacts may be of 
an indirect rather than a direct nature. These indirect impacts may take 
considerable time to manifest themselves, which suggests the potential 
importance of alumni or graduate surveys for assessing institutional im- 
pact. Often college has impacts which are not realized by graduates until 
some time considerably after they leave the institution. It is important to 
be sensitive to alumni and graduates as an important source of data 
concerning the long term effects of college or a particular institution. 



Conditional Effects 

Most attempts at value-added assessment at the institutional level as- 
sume that the impacts of educational experiences are general. That is, 
they assume that everybody benefits from the same experience in the 
same way. This assumption certainly has the appeal of parsimony (i.e., 
other things being equal, the simplest explanation is often the optimal one). 
It can be argued, however, that assuming only general effects in one’s 
analytic or assessment model ignores individual differences among stu- 
dents attending the same institution or exposed to the same educational 
or instructional experience. These individual differences among students 
may interact with different instructional, curricular or other educational 
experiences to produce conditional rather than general effects. Thus, the 
magnitude of the influence of certain educational experiences on student 
development may vary for students with different characteristics (e.g., 
level of entering aptitude, degree of prior exposure to, or competence in, 
specific course content, level of intellectual orientation). Conditional rela- 
tionships such as this might well be overlooked in assessment approaches 
which consider only general effects. In certain situations this may lead 
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one to conclude that effects of specific educational experiences are trivial, 
when in fact they may have pronounced positive effects for certain sub- 
groups of students. In my own research, for example, I have found that 
the effects of self-paced, mastery instruction on mathematics learning 
among freshmen has its greatest positive impact on the least well pre- 
pared students (Pascarella, 1978). 

The idea of conditional effects may run somewhat counter to state-ini- 
tiated mandates for accountability in which institutions are expected to 
demonstrate certain levels of effectiveness in promoting cognitive and 
other development for all students. Nevertheless, the consideration of 
conditional effects might well function to sharpen the focus of value-added 
assessment at the institutional level and enable it to better identify those 
particular students who are benefiting most or least from certain educa- 
tional experiences. This information could then be used to focus institu- 
tional efforts on those student constituencies where its efforts appear to 
be least effective. 



Qualitative/Naturalistic Approaches 

Sometimes of late when I am computing what must be at least my 
10,000th multiple regression equation, I have begun to wonder if I am 
really getting at the essence of college impact. I am beginning to think 
that what quantitative, logical positivists such as myself do best is to log 
out a broad map of the territory. It may well be that in the next 20 years 
it will take qualitative or naturalistic inquires to fill in all the rich and 
important topographical detail, which is often missing in quantitative 
analyses. Qualitative or naturalistic approaches, which employ interviews 
and direct observation, may be an important new direction for under- 
standing the ways in which college enhances student development. Such 
approaches are important in triangulating or verif5nng the results of more 
broad-based, traditional quantitative assessment; yet they may also be an 
important primary data source in their own right. 



Existential Outcomes 

Often assessment of outcomes is limited to what is posited as the “edu- 
cational impacts” of college (e.g., critical thinking, content knowledge, per- 
sonality and value changes). In doing so, the more existential outcomes 
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are often ignored (e.g., how enjoyable and satisfying are the years spent 
in college). If one lives to be 72 and obtains a bachelor’s degree, he or she 
will probably spend about 5 years more or less on a college campus. That 
is a considerable part of one’s life, roughly 6-7 percent of it. This suggests 
that an important part of assessment in postsecondary education is the 
attempt to understand the quality of student life at one’s institution. Do 
colleges amd universities succeed in making the time spent on their cam- 
puses as enjoyable and productive as possible? While it is often overlooked 
in the concern for measuring “educational impacts,” I would argue that 
the quality of campus life is a legitimate concern in assessing the benefits 
of college. 
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There can be little doubt that “assessment” is here to stay. At least 
seven national reports have appeared in the last five years, all critical of 
higher education in America and all giving a central role to “assess- 
ment”— the measurement of the educational impact of an institution on its 
students. Eleven states have adopted formal assessment requirements 
(Ewell, 1987), as many more are moving in that direction, and regional 
accrediting associations are tr5ring desperately to recover some of the 
accountability thunder they have lost to state governments. 

The fact that the origins of the push toward assessment are external 
to most campuses is significant. Surveys indicate that while “over 50 
percent of college administrators support assessing general education, . . . 
only 15 percent report doing anything about it. In the more complex area 
of ‘value-added’ assessment, some 65 percent support the concept but less 
than 10 percent are fielding value-added programs” (Ewell, 1987, p. 24). 
The clear implication of such figiires is that for many colleges and univer- 
sities, assessment is a relatively new undertaking: they are either just 
beginning to explore and implement assessment programs, or they have 
not yet even begun. In fact, most campuses have been engaged in assess- 
ment in a variety of areas. The difference is that these efforts have been 
undertaken unilaterally by various individual offices or groups, are not 
coordinated in any way, nor are they part of any overall institutional plan 
for on-going self-study. 

The current assessment literature is replete with descriptions of the 
benefits of assessment to institutions and students. Those benefits are not 
trivial, but because they are detailed in lots of other places, they are only 
suggested here. One clear benefit of assessment is that it forces institu- 
tions to focus on students. For years, as Sandy Astin (1985) has pointed 
out, colleges and universities have equated institutional and program- 
matic quality with the resources invested in their institutions. The “best” 
colleges and universities are the ones who have students with higher 
aptitude and achievement scores, more books in the library, more faculty 
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with terminal degrees, lower student-faculty ratios, and so on. It is well to 
move away from such measures. 

A second benefit, related to the first, is that assessment requires the 
paying of attention to the outcomes of college, to what colleges are doing 
to and for students, not to the resources that colleges are generating or 
expending. Third, assessment attaches a high value to growth, to student 
progress toward some set of specified educational objectives. These are all 
substantial benefits, and if assessment is seen as merely one more exter- 
nal reporting obligation, a great opportunity to enhance educational pro- 
grams will be missed. 

But while advice on how assessment programs should be designed and 
implemented is easy to come by, the pitfalls of assessment are more 
obscure, ts^pically treated only cursorily in the literature, if they are 
treated at all. This paper will call attention to some of those pitfalls and 
suggest, however briefly, how at least some of them might be avoided. It 
is NOT the intention to discourage institutions from developing assess- 
ment programs, but first, to identify some of the serious conceptual, meas- 
urement, organizational and political problems likely to be encountered in 
the process of designing and implementing an assessment program, and 
second, by identif 3 dng some of the pitfalls, to help people who are involved 
in assessment to do it well. To accomplish these purposes, three major 
areas will receive focus: (1) definitional issues; (2) organizational and 
implementational issues, and (3) methodological issues. 



Definitional Issues 

One of the most significant and imposing obstacles to the advance- 
ment of the assessment agenda is the absence of any consensus on pre- 
cisely what assessment means. To some, the term means testing individ- 
ual student achievement levels in various academic areas. To others, it 
means a review of the general education program and an evaluation of 
whether students are receiving a “liberal education.” To still others, it 
means a series of alumni surveys designed for program evaluation and 
planning purposes. And for still others, it means nothing less than institu- 
tion-wide self-study, applicable to teaching, research, service, and admin- 
istrative and management functions. 
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In thinking about what assessment is, it is useful to keep two ques- 
tions in mind, for the answers to those questions will have a powerful 
influence on the kind of assessment one becomes involved in and on the 
issues and problems one will face. The first question is: “What is the focus 
of the assessment?” This is the “purpose” question. Why is the assessment 
program being designed? What is it intended to do? At some risk of over- 
simplifying the matter, answers to this question generally fall into one (or 
both) of two categories: assessment for the enhancement of teaching and 
learning, or assessment for purposes of accountability to some external (or 
higher) body. In a sense, the answers to this question parallel the issues 
in formative and summative evaluation: the first is intended to inform 
change and guide program modification, while the second is to reach some 
final judgment about worth or value. 

The second question is: ‘What is to be the unit or level of assessment?” 
Is the assessment to focus on individual students (as “individuals,” not as 
elements in a group)? Or on groups, where “groups” can refer to aggrega- 
tions of students at the course, program, department, college/school, cam- 
pus, or system level? If these two questions (and the answers to them) are 
juxtaposed in a simple two-by-two matrix, it begins to become evident 
how var3dng forms of assessment can be categorized. 

Approaches to assessment with a clear “accountability” orientation 
can be of two varieties. If the assessment is at the individual level, the 
assessment performs something of a “gatekeeping” function, sifting and 
sorting the qualified from the imqualified. Assessment activities within 
this category include such practices as admissions testing (e.g., ACT, 
SAT, and others) and “rising junior” examinations employed in Florida, 
Georgia, New Jersey and elsewhere. Other varieties of the accountability- 
oriented, gatekeeping conception of assessment include comprehensive 
examinations in the students academic major field (a practice enjoying a 
revival) and certification examinations in professional fields (e.g., nursing). 

These latter sorts of examinations may also serve assessment pur- 
poses at the group level, however, where the interest is in group means 
rather than individual scores. At this intersection of the focus and level 
issues, the principal interest in assessment is for program enhancement, 
to determine the level of effectiveness or quality at which a program, 
department, school or entire campus is functioning. Assessment activities 
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in this cell include academic program reviews, analysis of student attri- 
tion rates and reasons, alumni follow-up studies, and “value-added” as- 
sessment programs. The focus or purpose is primarily evaluative, and the 
information so obtained may he used both for accounting to external 
bodies and for internal program improvement and planning. When used 
for program enhancement, these activities slide from the “accountability” 
side of the matrix to the “learning” side. (Just remember: in assessment, 
very few things are clear cut!) 

When the “focus of assessment” is on learning, and when assessment 
is at the individual student level, some would assert that as dealing with 
assessment in its purest form. In this approach, most notably practiced at 
Alvemo College, analysis of individual student performance is seen as an 
integral part of the teaching and learning process. Teachers use such 
information on student knowledge and skill mastery not only to guide 
individual student leEuming, but also to shape their teaching strategies 
and styles so as to maximize the learning potential of their classes and 
assignments. Feedback is fundamental. Indeed, without it, learning-ori- 
ented assessment is meaningless. 

Some other standard assessment practices also fall in this category, 
however. For example, placement examinations and other diagnostic 
measures intended to identify learning readiness and permit assignment 
of the student to the most beneficial learning sequence clearly have both 
an individual student and learning orientation. 

In addition to understanding why the wish to assess something, it is 
equally critical to be clear about what a college is interested in assessing. 
Several “outcomes” taxonomies have been suggested (e.g., Bowen, 1977; 
Pace, 1979; Leaning, 1979). A simple yet useful general typology has been 
given by Ewell (1984), who suggests four basic dimensions of outcomes: 
(1) knowledge outcomes; (2) skills outcomes; (3) attitudes and values out- 
comes; and (4) behavioral outcomes, particularly behaviors related to oc- 
cupations, post-baccalaureate educational activities, and professional and 
community activities. An institution must, of course, decide for itself 
which of these dimensions, or even subsets within these major categories, 
should receive primary attention. 

The point of all this is the importance of being absolutely clear, on a 
more-or-less campuswide basis, about what the assessment program is 
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intended to accomplish. Time spent in committee work and other forms of 
public discussion of the purposes and objects of assessment on a campus 
will be time extremely well spent. An inadequate programmatic and po- 
litical foundation for the assessment program will produce confusion, 
anxiety, and more heat than light. 



Organizational and Implementational Issues 

Assuming some reasonable level of agreement is reached on the pur- 
poses and objects of assessment, it is important to keep always in mind 
that institutional change is embedded in any conception of assessment. 
Depending upon where the changes occur, and how they are managed, 
they can produce higher levels of individual and organizational perform- 
ance and pride in accomplishment, or they can produce internal insurrec- 
tion. Institutional researchers and others must understand and appreci- 
ate that several significant organizational and implementational hazards 
must be addressed at the outset. 

A vital and difficult area involves mobilizing support. The endorse- 
ment and visible support of senior executive officers (including the presi- 
dent) is absolutely necessary, but, unfortunately, not sufficient. Faculty 
support is also needed, and without it prospects for a successful assess- 
ment program are dim. According to Ewell (1984), faculty objections are 
likely to come from either or both of two sources; first, the fear of being 
negatively evaluated, and second, a philosophical opposition based on the 
belief that the outcomes of college are inherently unmeasurable and that 
the evidence from such studies is “misleading, oversimplifying, or inaccu- 
rate” (p. 73). 

If sufficient attention has been given to public discussion and review 
of the program’s purposes and objects, much will already have been done to 
allay faculty fears. Assessment— even when required by an external body- 
must be, and be seen by all as, a vehicle of individual and institutional 
improvement, not a search for documentation on which to dismiss individu- 
als or retrench programs. Some basic leyel of trust must be established. 

Ewell (1988) recommends being publicly clear about what an assess- 
ment program is not intended to do. This would include clear specification 
of what data are to be collected, by whom, for what purposes, the conditions 
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under which the data will he made available, and to whom they will he 
available. Northeast Missouri State University (1984), which has had 
extensive experience with assessment, recommends that assessment data 
not be used to support negative decisions. 

Faculty reservations about the measurability of outcomes must also 
be addressed, and several approaches are possible. One is to acknowledge 
publicly the inadequacy of any single outcomes measure emd to ensure 
that multiple measures are incorporated into one’s assessment program 
(Ewell, 1984). It must be remembered that judgments of program emd 
institutional quality are made all the time, by a lot of different people. 
The issue is not really whether assessments should be made, but the 
nature emd sources of the evidence to be used to make those judgments. 

Another useful suggestion is to involve faculty members in the design 
of the assessment process (Ewell, 1984). Indeed, faculty members on most 
campuses constitute a significant, untapped source of technical, as well as 
political, support for an assessment effort. Similarly, it is useful emd wise 
to distinguish between “emalysis” and “interpretation” of assessment re- 
sults (Northeast Missouri State University, 1984). Analysis involves a 
series of technical activities, best performed by emalytically competent 
persons. Interpretation, however, involves judgments about meaning and 
should involve broad participation, particularly by those most directly 
affected. Persons involved in the interpretation of results emd decisions 
about alternative courses of action are far more likely to support those 
actions than are persons excluded from the interpretation emd recommen- 
dation process. 

Finally, assessment programs that start small, perhaps on a pilot 
basis, are more likely to draw support than gremd efforts with consider- 
able apparent potential for campuswide disruption. Most successful pro- 
grams begem small and grew incrementally (Ewell, 1988). Northeast Mis- 
souri State University, for example, began its assessment efforts in 1973. 
And an inventory of current data collection activities (including the use of 
standardized measures, program review results, surveys, and stemdard 
institutional research studies) is likely to reveal greater current involve- 
ment in assessment than might at first have been believed. 

In short, one cannot overstate the importance of laying a strong politi- 
cal foundation. Without it, the structure cannot stand. And it is well to 
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keep in mind that some of the most important benefits of assessment 
come from the process of self-analysis, not just from the results. 

A second major issue concerns coordination. Assessment requires the 
involvement of a wide variety of people and offices, crossing not only 
academic departmental lines, but vice presidential areas as well. Alterna- 
tive approaches to the campuswide coordination problem have included 
assignment of coordinating responsibility to a currently existing office 
already significantly engaged in assessment (and controlling many of the 
necessary resources), creation of a new office, or assignment to a commit- 
tee with representatives from the major affected organizational areas 
(Ewell, 1988). Each of these approaches, of course, has its assets and 
liabilities, and while space will not permit a detailed discussion of each, 
the reporting line(s) for the office or group should be given careful atten- 
tion. Whatever approach is adopted, what will be the likely effects on 
traditional areas of responsibility and lines of authority? On informal 
power networks? On traditional distinctions between academic and stu- 
dent affairs? Ways will have to be found to coordinate activities such that 
lines of authority and responsibility are clear, existing functions and ac- 
tivities are not duplicated, and support is received from each area. Experi- 
ence indicates that the support of the institution’s top executives, particu- 
larly the president and academic vice president, must be active and visible, 
particularly in the early stages of the program’s development (Ewell, 1988). 

Finally, one cannot ignore the costs of assessing educational outcomes. 
Assessment costs are little understood, and it is precisely because of igno- 
rance in this area that it constitutes such a potential threat to the success 
of any campus-based program, as well as the general national trend to- 
ward assessment. Much has been written about the “benefits” side of the 
cost-benefit equation, but only recently has attention been turned to the 
cost side. How much should an institution invest in its assessment pro- 
gram? The question is unanswerable, of course, in the absence of consid- 
erable knowledge about purposes, activities and areas of assessment. But 
Ewell and Jones (1986) have argued that the real question is one of 
marginal costs: “How much more money (beyond that already committed 
to outcomes-related information gathering) do we have to spend to put in 
place an assessment program that is appropriate to our needs?” (p. 34). 
These costs are incurred in four areas: (1) instnunent costs, (2) admini- 
stration costs, (3) analysis costs, and (4) coordination costs (Ewell & 
Jones, 1986). After making a series of assumptions about the nature of 
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the assessment program likely to be mounted by institutions of varying 
types and sizes, Ewell and Jones (1986) estimate incremental costs rang- 
ing from $30,000 (in a small, private, liberal arts college) to $130,000 (in 
a major public research university). It is important to bear in mind that 
these are incremental costs, to be added to already-existing data gather- 
ing and analysis costs. Moreover, they do not include personnel costs 
associated with faculty involvement in assessment. And if one assumes an 
on-going assessment program (which is strongly recommended), the Ewell 
and Jones’ estimates are also annual incremental costs. 

Finally, opportunity costs must also be considered. Institutional re- 
sources (including time) invested in assessment are not available for in- 
vestment elsewhere. Moreover, Governor Kean (1987) of New Jersey has 
urged institutions not to ask for additional funds to cover assessment 
costs. He does not believe legislatures will respond favorably to requests 
for money to determine whether past and current appropriations are 
being effectively utilized. If that is true, reallocation of currently appropri- 
ated funds will be necessary. 

Methodological Issues 

The third category of potential pitfalls is methodological. Some are 
specific to particular approaches to assessment; others are merely com- 
mon and frequent violations of the canons of good research. Within this 
general area, problems fall into three sub-categories: (1) design problems, 
(2) measurement difficulties and (3) statistical hazards. 

Design Problems: One of the dominant themes in the calls to assess the 
educational outcomes of college, and the primary thrust of the demands 
for “accountability” through assessment, is the need to demonstrate that 
college makes a difference, that colleges are in fact delivering on the 
claims that fill admissions materials about the benefits of college atten- 
dance, that students leave colleges and universities with knowledge, 
skills, attitudes and values they did not have when they arrived. 

The fact that students change in a variety of ways between their 
freshman and senior years has been demonstrated in an impressive num- 
ber of studies (e.g., Bowen, 1977; Pace, 1979). The design problem lies in 
determining the origins of those changes, which may also be due to the 
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pre-college characteristics of students, or to growth processes and experi- 
ences that are quite independent of college. Thus, collegiate impact is only 
one of the possible sources of freshman-senior year change. It may be a 
significant source of change, but knowing with any degree of certainty 
whether and to what extent college has an effect is a very complicated 
matter. 

A common approach to the assessment of change in students is the 
use of successive cross-sections designs. Typically, cross-sectional samples 
of current freshman and senior students are taken and the two groups are 
compared on some measure of the variable on which change is being 
studied. Such designs, however, assume that current seniors, at the time 
they matriculated, were similar in important respects to current fresh- 
men— a questionable proposition. Such designs also leave selective dropout 
during the college years uncontrolled. Not all students who begin college 
will finish it. In such a circumstance, freshman and senior group means 
would be different, even if the two groups had been identical at the time 
they both entered school-also a doubtful proposition. 

Longitudinal designs are a frequently recommended alternative. At 
least some of the same people are being studied at the two different times, 
but note the emphasis on the word “some.” Subject mortality (the ten- 
dency of subjects to drop out of a study over time) can be a significant 
problem with longitudinal research, particularly those that cover an ex- 
tended period, like four years. As response rates drop, study generalizabil- 
ity is threatened. 

Moreover, because of this subject mortality problem, longitudinal de- 
signs require larger samples, which increase costs and data management 
requirements. It is important to keep in mind that research design is a 
series of compromises. Designs that increase the power of a study in one 
area almost invariably come at a cost somewhere else in the study. When- 
ever something is gained, something else is given away. The point is to 
know what is being gained, and what is being given away (see Terenzini, 
1980). 

Most assessment studies are, of necessity, single-institution studies, 
and even when based on longitudinal designs, there are important con- 
straints on what they can and cannot do. In the zeal to demonstrate the 
beneficial effects of attending a college or university, other powerful forces 
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that influence students during the college years are easily forgotten, nor- 
mal maturation probably being the most significant one. In single-institu- 
tion designs, it is just not possible to separate the effects of college from 
those of maturation. Instrument decay and regression effects (the ten- 
dency for the scores of persons with very high or very low initial scores to 
move toward the mean on retesting) may also be involved, but maturation 
is the most likely alternative to college impact as an explanation for 
change observed in college students. Pascarella (1987) has suggested sev- 
eral useful ways to deal with this problem. 

In developing a sample plan, it is absolutely crucial to have clearly in 
mind the kinds of sub-group analyses that one might wish to conduct. As 
the niunber of groups grows large, or as one or more of the subgroups 
comprise a small population (e.g., minority students), simple random sam- 
pling plans are likely to be inappropriate. 

It has been frequently recommended, for example, that successful as- 
sessment programs provide unit-specific information. It is easy for deans 
or department heads to disregard assessment information when it is not 
specific to their college, school or department. The implication of this 
advice, however, often overlooked until it is too late, is that a census, not 
a sample, of students must be taken. Otherwise, group sizes may be too 
small to have face validity, political believability, or statistical stability. 
Costs and workload will go up accordingly. 

Measurement Problems: When discussing “measurement,” the issues are 
reliability and validity, questions that are always at issue in the selection 
of an instriunent. The common dilemma, of course, is whether to adopt a 
commercially-available measure (e.g., the ACT’s COMP, or measures now 
available through ETS’s Program Self-Assessment Service), or to devise 
an instrument locally. As noted previously, research is one compromise 
after another. Adoption of a nationally-available measure has several 
advantages. First, it was developed by experts in measurement. Second, 
it has been field-tested and its psychometric properties are known. Third, 
national scores or norms are probably available so one can compare one’s 
own institution with others of similar size, type and purpose. Finally, they 
can save substantial amounts of time and expense that would otherwise 
go into local instrument development. 
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Such adv^tages come at a cost, however. Commercial measures, in 
order to be useable in a variety of settings and for a variety of purposes, 
are necessarily general and lack the specificity needed to focus in any 
detail on local conditions. Similarly, standardized achievement measures 
must necessarily focus on a limited number of learning objectives which 
may or may not be the ones a campus has set for its students. While 
locally developed measures may be more carefully tailored to local pur- 
poses, they are also likely to be untested (at least in the short run) and, 
consequently, of unknown reliability and validity. Moreover, construction 
of instruments is not an activity for novices. 

Validity can also be increased through the use of multiple measures. 
Do not rely on single measures or types of measures (e.g., surveys, or 
objective tests). Unobtrusive measures, ones which do not intrude on the 
subjects’ awareness of being studied, can be highly useful (see Terenzini, 
1987). 

One of the most commonly used measures in assessment programs is 
the College Outcome Measures Project (COMP) examination of the 
American College Testing Program (ACT), one of the few commercially- 
available measures of knowledge and skills presumably imparted by a 
general education program. In assessing the effects of such programs, 
most institutions are unable or unwilling to wait two to four years in 
order to follow a cohort of entering freshmen through to their senior year, 
as would be necessary in a longitudinal research design. ACT has re- 
sponded by providing an estimate of student gain in the COMP total score 
(Banta, Lambert, Pike, Schmidhammer, & Schneider, 1987). Based on a 
comparison of the COMP and ACT Assessment Composite scores of stu- 
dents who have taken both tests, ACT has developed concordance tables 
which permit institutions to “estimate . . . the (COMP) score gain, or 
value-added” (Banta, et al., 1987, p. 3). 

Banta and her colleagues (1987) have conducted a series of tests on 
these estimated gain scores, and their findings are noteworthy, not to say 
astounding. Among them are the following; 

1. Because of the large standard deviation of the estimated gain, “in 
a given year the level of estimated gain for (University of Tennes- 
see— Knoxville students) could be in-error by as much as 20 percent.” 
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2. The regression equations upon which the concordance tables are 
based differ significantly from year to year, and estimated gains 
for UTK students were underestimated by as much as 60 percent! 

3. Estimated gain scores are change scores, and they are notoriously 
unreliable. Indeed, Banta and colleagues concluded; “the level of 
dependability of estimated gain is simply too low to serve as the 
basis for making decisions about program quality or the allocation 
of resources” (Banta, et al., 1987, p. 10). 

4. A large number of graduating seniors had no ACT Assessment 
Composite score from which a COMP estimated gain score could be 
calculated. Those without Assessment scores tended to be older, 
black, and from lower socioeconomic fiimilies, raising serious ques- 
tions about the generalizability of any study based on estimated 
gain scores. 

5. Finally, the correlations between all estimated gain scores and 
certain demographic and institutional variables were the opposite 
of what was expected. For example, the greatest gain scores 
tended to be those of students who had high school averages lower 
than 3.0, did not receive a scholarship, whose fathers did not 
graduate from college, who did not participate in honors math 
sections, and who did not take more than two math courses 
(Banta, et al., 1987, p. 15). 

Clearly, these results raise serious questions about the reliability and 
validity of all estimated gain scores, not just those produced using the 
act’s comp and Assessment instruments. Indeed, Banta and her col- 
leagues are quick to praise COMP as a “valuable tool for stimulating 
faculty discussion about the general education curriculum, and modes of 
instructions. . . What is called into question is the usefulness, the validity, 
of employing estimated student score gain on the COMP for the purpose 
of making precise judgments about prograim quality that can serve as the 
basis for decisions about the allocation of resources in higher education” 
(p. 18). 

One might infer from this that the use of actual gain scores is a way of 
circumventing the problems Banta and her colleagues identified in esti- 
mated gain scores. However, even actual gain scores must be used pru- 
dently. 
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Statistical Problems: As noted previously, assessment, particularly the ac- 
countability strain, has embedded in it the expectation that change will 
occur, that the institution’s contribution to student learning can be made 
apparent, even measured with some precision. Unfortunately, almost 
without exception, researchers rely on average changes, or a comparison 
of a group mean at Timel with the same group’s mean at Time2. But 
group chgmge often masks individual change. Any observed freshman- 
senior chgmge is related to the number of students who change and to the 
amount of change each student experiences. It may be useful to give 
attention to the frequency, direction and magnitude of individual change 
(see Feldmgm & Newcomb, 1969, pp. 52-69). 

Moreover, change is often construed as value-added, a frequently 
heard phrase that can be terribly misleading and damaging if not under- 
stood. Warren (1984) gmd Pascarella (1987) offer thoughtful and detailed 
discussions of this concept, but certain aspects of it require attention here. 

Value-added is both a metaphor and a research design. As a meta- 
phor, it is highly vivid and useful in focusing attention on institutional 
effects, rather than on institutional resources. “Quality” is seen as a func- 
tion of what is “produced,” not of the number of volumes in the library, 
percentage of faculty with the terminal degree, student-faculty ratios, 
instructional expenses per student, and so on. Unfortunately, it can some- 
times be too vivid, leading people inside and outside the academy to 
expect more of assessment programs than can possibly be delivered. 

The reason for this lies not only in the metaphor’s assumption of the 
occurrence of “change,” but also in its assumption of positive change or 
growth. Can “value” be “added” without positive change? Legislators and 
others are likely to say “No.” And therein lies the perniciousness of the 
metaphor, for it is important to distinguish “change” from collegiate “im- 
pact.” As Pascarella (1987) notes: “In some areas of development . . . the 
impact of college (or other educational experiences) may be to prevent or 
retard decline rather than to induce major positive changes. Conse- 
quently, approaches to value-added assessment which focus only on pre- 
to post-chgmges may be overlooking important college effects” (p. 78). 

The value-added metaphor also promotes a research and analytical 
design that is correspondingly simple and equally pernicious. To know 
whether something changed, measure it at Timel and again at Time2. 
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The difference between them, the change score, reflects the institutional 
value-added. Right? Wrong! The interpretive problems when maturation 
effects remain uncontrolled have been discussed earlier. But reliance on a 
simple difference score has other problems. Difference scores are known to 
be notoriously unreliable. Indeed, they have some positively alarming 
characteristics. For example, it can be shown that difference scores are 
negatively correlated with pretest scores (Bereiter, 1963; Linn & Slinde, 
1977). This negative relation lies behind the counterintuitive, even bi- 
zarre, findings reported by Banta and her colleagues (Banta, et al., 1987). 

Second, it can also be shown that the higher the correlation between 
pre- and post-test measures, the lower the reliability of the gain score 
(Linn & Slinde, 1977), making detection of reliable associations with 
other variables (e.g., aspects of the institutional experience thought to 
produce a portion of the change) more difficult. 

Third, simple difference scores are also subject to ceiling effects: stu- 
dents with high pretest scores have little room for improvement and, 
thus, are likely to show smaller “gain” scores than students with lower 
initial scores. Similarly, gain scores are subject to regression effects, the 
tendency— due strictly to measurement error— for initially high (or low) 
scores to move (“regress”) toward the group mean upon subsequent testing. 

Thus, in considering the use of the value-added metaphor in a specific 
measurement setting, we would be well advised to follow the suggestion 
of Cronbach and Furby (1970) to “investigators who ask questions regard- 
ing gain scores.” That advice is to “. . . frame (our) questions in other 
ways” (p. 80). Linn and Slinde (1977) and Pascarella (1987) suggest a 
number of those ways, although we cannot review them here. But there 
should be some comfort in the knowledge that it is possible to assess 
institutional impact in ways that are conceptually understandable and 
methodologically preferable to the use of simple difference scores. 



Conclusion 

The assessment of student outcomes is a complicated business. Most 
really important things usually are. Do not let the complexity deter the 
effort. Both the processes and products of assessment have much to offer 
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colleges and universities. Indeed, in many respects, assessment is a prac- 
tice colleges and universities should have been doing all along. Product 
evaluation is really just part of good management. At the same time, 
however, assessment is not something that can be done quickly or casu- 
ally. It is complicated enough without having a number of serious concep- 
tual, administrative, political and methodological briar patches along its 
path. 
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INSTITUTIONAL EFFECTIVENESS 



James R. Montgomery 

Virginia Polytechnic Institute and State University 



In about the first or second session of einy course on management or 
administration, the instructor introduces the topic of scientific manage- 
ment emd the name Frederick Taylor. Taylor’s approach led to the belief 
that operations research emd systems memagement coiild improve deci- 
sion making, aid in the management process, and thereby improve insti- 
tutional effectiveness (Woolf, 1965). 

Following World War II the so-called GI Bill encouraged service men 
and women to return to colleges and universities. A baby-boom of that era 
soon sparked higher enrollments in the 1960s. In addition, increasing 
numbers of youth emd their parents believed that a college education 
woiild be useful emd made finemcial sacrifices in order to obtain it. Nation- 
ally, total enrollments in postsecondary education soared over 430 percent 
from 1950 to 1980 (Center for Statistics, 1985-86, 1970). How these na- 
tional numbers translate into those for one lemd-gremt iiniversity (Vir- 
ginia Polytechnic Institute and State University) are described in Table 1. 

Table 1. Enrollment Increases in Postsecondary Education 





NATIONAL 


LAND-GRANT 


Year 


Number 


Percent 


Number 


Percent 


1950 


2,281 ,298 




3,781 




1960 


3,582,726 


57 


5,106 


35 


1970 


8,580,887 


140 


12,043 


136 


1980 


12,096,695 


41 


21,069 


75 



Such enrollment figures suggest that colleges emd iiniversities that 
wemted to grow emd had sufficient drive by management and facility 
could do so. Moreover, such growth might be quite rapid and challenges 
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existing managers to adapt to changing times. Many existing institutions 
doubled and then again doubled in size; new colleges sprang to life. The 
community college sector dramatically increased opportunities for those 
who had not considered college or who wanted an alternative to the usual 
collegiate format. In the community college, all students went through 
the same front door-some tirrning to the right for a vocational-technical 
coirrse and others to the left for a more traditional academic and colle- 
giate coirrse; all of them could claim the same college, and their diplomas 
looked remarkably alike. 

Turning to economic theory, economies of scale state that as produc- 
tion increases, average overhead could be reduced and, therefore, more 
profit made. Seen in the collegiate light, a college could double student 
enrollment with an increase from 200 to 400, and add only faculty mem- 
bers and perhaps a department head. What would happen, however, if 
enrollment doubled from 1,000 to 2,000 students? Could the college keep 
the same administrative structirre and add about 66 new faculty mem- 
bers? Maybe, but the answer becomes harder to predict. Could a college 
increase from 5,000 to 10,000 students and add only faculty? No! Re- 
search has revealed that as institutions grew economies of scale were 
overpowered by increasing complexity. New faculty members wanted to 
teach material studied in graduate school, and new coirrses were added; 
new specialities proliferated in the computer and space age through which 
the nation was passing; research organizations sprang into life; it became 
necessary to add new departments, then new deans or faculties, and then 
a vice president for academic affairs or provost and several assistants and 
associates to oversee the complexity developing. Large institutions could 
benefit from economies of scale only to a point, for complexity more than 
offset it (McLaughlin, Montgomery, Smith, Mahan, & Broomall, 1980). 



Institutional Research 

Against the backdrop of scientific management, increasing enroll- 
ments, and complexity, college administrators needed more information 
for decision making and new tools to use in administration. Among the 
pioneers in bringing new techniques to postsecondary education were two 
who made a difference. John Dale Russell (Russell & Doi, 1955, 1957) 
collected information on statewide systems, for example higher education 
in Virginia or Michigan, and on how to derive usable figures to review 
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institutional expenditures or space needs on a college campus (highly 
useful information with the construction projects needed to keep pace 
with enrollment). A.J. Brumbaugh (1960) wrote a small booklet that the 
American Coimcil on Education (ACE) distributed widely. He argued that 
colleges amd universities could improve management by adding staff to 
collect amd amalyze data for management decision making. The publication 
had an imusually high impact on colleges and imiversities. Thus, Russell 
and Brumbaugh urged management to use more data and better orgam- 
ized information in decision making. Scientific mamagement had found 
allies amd would thereafter be pressed forward on many college campuses. 

With the urging and encouragement of the ACE, Southern Regional 
Education Board, Southern Association of Colleges amd Schools (SACS), 
Western Interstate Commission for Higher Education, amd other 
orgamizations, the field of institutional research was launched. It was 
then, amd is today, nothing more complicated tham stud 5 dng am institution 
to see how it works and providing information that might prove useful to 
improve or change the direction of management. 

At about the same time that institutional research was developing, or 
shortly thereafter, mamy institutions added a planning fimction to help 
gmde institutional development. In addition, in the 1960s the small 
trickle of data from the oversized amd inefficient computers soon became a 
raging torrent as increasingly sophisticated computers poured out data 
for management to decipher. 

Scientific management as applied to the college campus did not go as 
far as that used in the corporate model, and it never was called by such a 
title. But scientific management had nevertheless permeated the post- 
secondary environment, and it became a factor in institutional effectiveness. 

Reputational Studies 

What makes an educational institution good, or outstanding? Stu- 
dents who graduate should have acquired a depth of knowledge in a field; 
they should be competent. But words like competent, depth, good, and 
outstanding have var 5 dng interpretations. Therefore, how to evaluate ef- 
fectiveness, what to evaluate, and what constitutes good or outstanding, 
remain issues of concern for colleges £md imiversities. In the United 
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States, wide agreement exists that Harvard or the University of Califor- 
nia at Berkeley are outstanding universities; however, in the United 
States there are over 3,300 other colleges and universities, meiny of which 
perform competently an educational mission. Will the farm boy or girl 
really receive from Harvard a better education for futirre life on the farm 
than at a land-gremt college? Will the irrbem youngster working in the 
daytime and studying at night really be better served by a resident col- 
lege them by the nearby irrban college that opens its doors every night 
and on the weekends? The answer to both questions is no, and that an 
institution, by virtue of different missions emd resources, can make an 
impact in its chosen arena. An institution should be evaluated in light of 
those differing missions and resoirrces. 

A favorite evaluative procedure has been to use peer evaluations of in- 
stitutions or disciplines (departments) within an institution. These 
reputational studies have appeared at var 5 dng intervals since 1924. Per- 
haps the best known were two conducted under the auspices of the ACE 
in 1966 and again in 1970. Called the Cartter emd Roose study and the 
Anderson study, the findings contain comparative listings of disciplines in 
slightly over a hundred major research universities. These studies sought 
to compare the institutions on faculty quality and effectiveness of pro- 
grams (Conrad & Blackburn, 1985). The reputational study may feed the 
vemity of faculty or students in disciplines identified; it does little, how- 
ever, to improve a program or to suggest shortcomings in disciplines or 
programs not named or selected in such a listing. 

Accrediting 

Voluntary accrediting arrived in the U.S. in the late 1890s and early 
1900s. Colleges and universities joined in regional associations to strive 
for self-regulation and self-assessment. The associations encomage self- 
assessment emd use peer review to obtain compliance with published 
standards (or criteria). While their published stemdards vary somewhat, 
all six regional accrediting agencies have requirements relating to institu- 
tional purpose and objective, orgemization emd administration, finemcial 
resources, physical plant, library and learning resources, student services, 
faculty, and educational programs (Kells, 1981). 
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The SACS has pressed two innovative programs in accrediting its in- 
stitutions. Both ideas pertain to improved operation of institutions, that 
is institutional effectiveness. 

One idea, the self-study, started around 1960. In a self-study, an insti- 
tution spends about 18 months reviewing its mission and its operations 
(e.g., as related to faculty, educational programs, library). Typically, com- 
mittees review areas that may pertain to the total institution, then col- 
lege and department committees review issues of concern to them. From 
these committees emerge recommendations that can serve as a long- 
range plem for an institution. At the conclusion of a self-study, a visiting 
committee of faculty and administrators from other institutions reviews 
the self-study, evaluates how well the institution has met the standards 
(criteria), and makes recommendations for the future. Since recommenda- 
tions of the visiting committee require a response (or the institution faces 
a reprimand or even loss of accreditation), the institution involved does 
give attention to these recommendations. With an institution reviewing 
its mission and programs and formulating directions for the future, plan- 
ning emd evaluation does take place. The key to a successful self-study is 
sufficient attention from top management to assure that resources and 
talent are identified to take part in the process, then sufficient attention 
by top and middle management and faculty to review recommendations 
and take steps to place them in operation (whenever feasible). 

Does self-study work? It depends on leadership, but a better ongoing 
mechanism to obtain orderly change has not been found. When budget 
cuts force radical reductions the mechanism may be effective, but disrup- 
tion typically occurs. 

The second innovation by the SACS (1986) is a new criterion that is 
required for accreditation. Called “Institutional Effectiveness,” this crite- 
rion calls for planning, evaluation, and provision for institutional re- 
search. The criterion is designed to encourage self-assessment by requir- 
ing planning and evaluation. 

The criterion mandates that an institution “establish adequate proce- 
dures for planning . . .” (SACS, 1986). This requirement caused a collec- 
tive tremor to run through southern institutions; planning occurs, but 
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documentation and tying it to mission and goals or otherwise having it 
formalized are frequently fragmented. At times procedures for planning 
exist but fall into disuse; at other times plans exist but appear unrelated 
to operations. The criterion also calls for evaluation. “The institution must 
define its expected educational results and describe how the achievement 
of these results will be ascertained” (SACS, 1986). These requirements 
should include: a clearly defined purpose appropriate to collegiate educa- 
tion, formulation of educational goals consistent with the institution’s 
purpose, development of procedures for evaluations of the extent to which 
these educational goals are being achieved, and use of the results of these 
evaluations to improve institutional effectiveness (SACS, 1986). It is also 
expected that “institutions should ascertain periodically the change in the 
academic achievement of their students” (SACS, 1986). 

Criteria for Accreditation, Section III, also mentions institutional re- 
search. Perhaps three sentences suffice to illustrate the point. 

Because institutional research can provide significant information 
on all phases of a college or university program, it is an essential 
element in planning and evaluating the institution’s success in 
carrying out its p\irpose. Institutions should assign administrative 
responsibility for carrying out institutional research. Institutional 
research should be allocated adequate resources, and those respon- 
sible for it should be given access to all relevant information 
(SACS, 1986). 

In the early days of self-studies, institutions typically engaged in 18 to 
24 months of frantic activity in preparation for them; the other 8 or 8 1/2 
years focused on other activity. It is no longer clear that such lapses are 
possible; planning and its evaluation and assessment of “academic 
achievement” appear to be ongoing operations subject to continuing activ- 
ity and review. 

With such encouragement to management and institutional research, 
it should come as no surprise that the 1987 conference^ of the Southern 
Association for Institutional Research attracted a record of nearly 300 
attendees; in the same year the international forum of the Association for 
Institutional Research attracted about 800. In contrast to a few years 
ago, scarcely a week passes without an advertisement in The Chronicle of 
Higher Education calling attention to a new opening or a vacancy in the 
field of institutional research. 
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Evaluations 

In addition to new requirements for self-studies, many managers also 
face the challenge of evaluation or assessment. Evaluation has received 
new emphasis by state legislators and state educational agencies either 
pressing for or requiring some form of evaluation. As a consequence, 
assessment has become the shibboleth of contemporary higher education 
in the United States. 

Assessment combines the testing of knowledge with the application or 
the using of that knowledge (Edgerton, 1986). (For example, combining 
knowledge about a computer or the theory of programming with actually 
writing a Fortran program and getting it to work.) In a recent book, A.W. 
(Sandy) Astin (1985) has focused on educational excellence; ways to meas- 
ure excellence remain murky at best. 

Evaluations are a part of all institutions. The total institution can be 
evaluated, as in a self-study; a student can be evaluated, as in a grade for 
completion of a course or by virtue of receiving a degree that signifies 
achievement of mastery of some body of knowledge. Many possibilities 
exist for such evaluations. 

External evaluations occur as other administrators, faculty or consult- 
ants provide some form of evaluation on an institution or program; a self- 
study combines both internal and external review; internal review may be 
conducted at any level within the institution (department, program, col- 
lege, etc.). Many reviews can be made relatively inexpensively, but most 
evaluations carry some resource requirements. In looking into an institu- 
tion, it is possible to distinguish between input measures (a traditional 
emphasis) and outcome measures (now receiving much attention in the 
U.S.). The distinction between these measures is important, for it deter- 
mines, in part, how an institution will look at itself. 

The following are items that should be considered in the review process; 

1. Information on the operation of an institution (In these comparisons, 
we look at an institution over time in relation to itself or to its peers 
[these are typical “input” measures].) (Items A-C from Astin, 1985, 
pp. 37-39): 
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a. Amount of endowment 

b. Expenditures per student 

c. Student to faculty ratio 

d. Average class size 

e. Faculty salaries 
f Library holdings 

g. Physical plant size and value 

h. Financial value of research grants and contracts 

i. Test scores or high school grade average or rank of freshmen 



2. Information on outcomes or success of students (Items A-C from 
Astin, 1985, pp. 43-44): 



a. 



b. 

c. 

d. 

e. 
f 

g- 

h. 



i. 

j- 



Students in Student Who’s Who; alumni in Who’s Who type list- 
ings 

Entrants in graduate or professional school 
Success of graduates 

Alumni satisfaction with educational experience 

Licensing exam scores 

Job placement rates/quality 

Employer satisfaction with graduates hired 

Reports from accrediting agencies 

Exit interviews of graduating seniors 

Grades of students (or grades given within a department or pro- 
gram) 



3. Information on change in students (Astin [1985] calls this talent 
development): 



a. Value-added”given what student knew at entry level, what does 
he know at graduation 

b. Competency testing-a check to ensure students have mastered a 
subject area (e.g., an examination between the second and third 
years to ensure ability to write a theme) 

c. Attitude changes 

d. Related conceptual skills (e.g., ability to reason, to listen, to 
solve problems). 
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Improving Institutional Effectiveness 

Management has sought to improve its operation by adding institu- 
tional research and planning functions. Whether the addition came from 
internal or external forces does not matter. Management needs such re- 
sources to respond to a need for information, for planning, for the evalu- 
ation of such planning, and for the assessment of academic achievement. 

The first essential element to improve institutional effectiveness is to 
ensure that the staff has basic skills or knows how to respond to the 
needs of management. Whether the staff handling planning, assessment, 
and institutional study receives the title institutional research or not is 
immaterial. The key element is that personnel to perform such a function 
(by some name) be in place. Many managers, however, are turning to 
institutional research as a means to have sufficient person-power to ac- 
complish planning and operations-research-t)rpe activities. 

The next step is to get in place a planning process that moves the 
entire institution in the same general direction. That is easier said than 
done, for it is necessary to balance the ability to move quickly in an arena 
where little or no planning has occurred and the long-term plodding nec- 
essary to pirrsue a goal or objective that remains slightly beyond an 
outstretched grasp (e.g., better salaries for faculty). 

Planning and the assessment of outcomes have become key factors in 
institutional health. Institutional planning requires a mission statement, 
that is the expectations of the institution, and long- and short-term direc- 
tions for achieving these expectations. Usually planning, when effective, 
has relevance to financial resources and day-to-day operations. As previ- 
ously noted, however, institutions typically have few if any plans for 
mission statements in place. Planning should provide the institution with 
a “systematic process” and “objective results” (Fincher, 1987). Several 
planning models exist. The institution will need to adapt any one selected 
to its milieu and the personalities involved. 

It is easy to become lost in the planning process when moving from a 
lofty mission and purpose statement and institutional goals to the direc- 
tions an institution or a department should follow in a one to three year 
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period. Regardless of the number of people who may be involved in the 
planning process, the outcome can always be subjected to second guessing 
and criticized by those who did not have a direct input on the direction 
selected for implementation. 

Tying the broad mission statement or institutional goal back into a 
department direction is both possible and useful. Table 2 is an example of 
such an effort and can be applied at any level in the institution. The 
process from purpose to assessment to purpose can be diagrammed as in 
Table 3. 

The missing ingredient in most planning is that of assessing “objective 
results.” The evaluation of planning frequently is subjective in nature. 
The problem is not so much the manner of its evaluation as the lack 
thereof The field awaits ambitious researchers to help direct our prac- 
tices and techniques in the evaluation arena (Hipps, 1982). 

The key point, however, is that an effective institution has a planning 
process in place. When it is there, it is evident. 



Table 2. Overall Mission to Unit Level 



Institutional Statement 


“Promoting personal and intellectual 
development” 


Department Statement No. 1 


Students will be able to write a theme 
paper 


Assessment of No. 1 


Each student, prior to senior (or 4th) 
year, will write an acceptable theme 


Department Statement No. 2 


Students will critique an essay for 
grammar, spelling, and punctuation 


Assessment of No. 2 


Each student will identify 80 percent of 
the errors 
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The assessment of student achievement can be pursued in many ways. 
While the output measures as demonstrated by Northwest Missouri, 
University of Tennessee, or Alverno College provide examples, other 
means and measures are possible. Msmy institutions have these already in 
place— it is a question of using the results of them. Here are examples: 

1. In the South, almost all colleges survey faculty and students as part 
of a self-study. Such surveys can be expanded and the results more 
fully used. 

2. Alumni surveys c£m be used to critique programs £md review prog- 
ress of graduates. 

3. Grading patterns of departments are available, as well as data on 
freshmen and trsmsfers. 

4. An exit survey of graduating seniors can be used to give a quick 
response to faculties, departments, or colleges. 

5. Employers can be surveyed to obtain ideas on graduates and their 
preparation for the work place. 



Table 3. Relationship of Planning and Evaluation 



Statement of Institutional Purpose 
Goals or Objectives or Directions (Inst.) 
Implement Operational Plans 
Evaluate Achievement 
Feedback of Results 
Use of Results 
Make Adjustments 
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Only a lack of resources and imagination limits the type of measures 
that can he used to review effectiveness. The idea is to collect indicators 
that can he used to point to the overall health of an institution, program, 
department, and so forth. An effort is needed to avoid being caught up in 
a new testing procedure or some program just because another institution 
has adopted it. What is needed is a sober review of what the institution 
wants to do and then look for the indicators that will tell how that pro- 
gram is going. 

A key point to remember, however, is that evaluations and the indica- 
tors that are used to obtain them are usually designed to bring change. 
The further removed the person or department or institution is from the 
evaluation and its results, the more difficult it will be to bring change. 
When Virginia Tech recently developed a plan for assessment of under- 
graduate academic achievement, the primary part of the plan looked to 
the department to develop evaluative procedures, the idea being that the 
farther faculty are removed from assessment of academic achievement 
the less likely for any change to occur. 

In addition to the evaluation of instruction or student achievement, 
another form of evaluation for institutional effectiveness examines a 
department. Many institutions routinely bring in outside experts (from 
other parts of the institution or from other institutions) to review the 
performance of academic departments. Much less likely to occur are re- 
views of nonacademic departments or offices. Those operations, which 
supposedly support the academic mission and assist management, may 
harbor conspicuous cesspools of poor administrative practices. 

Examples of formal evaluations of senior administrators are hard to 
find. For the most part, such evaluations are haphazard or altogether 
absent. Yet, what better way to change or improve institutional effective- 
ness than by helping managers to improve themselves? 



Summary 

What is institutional effectiveness? It is effective instruction for stu- 
dents, good nonclass educational opportunities, good use of resources, ef- 
fective management. In short, it is the ability to cause an institution to 
perform its mission effectively. Managers and faculty members continually 
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seek that condition. There is nothing new or magical about planning, 
evaluation, and institutional study to aid institutional effectiveness. 
Rather, there is a renewed emphasis, a clearer objective, and an external 
emphasis to encourage action. This emphasis of the 1980s on evaluation 
and assessment will be replaced in the 1990s or the twenty-first century 
by other imperatives! In the meanwhile, an emphasis on institutional 
effectiveness and the planning, evaluation, and institutional study neces- 
sary to bring it about can only help improve institutions. Institutional 
effectiveness will make colleges and universities better places. 
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The relatively recent growth of institutional research as a profession 
has in many regards paralleled that of higher education in general. Al- 
though scattered institutional reports and studies can be found dating 
back to the 1920s and 1930s, the majority of activities that today are 
called institutional research trace their roots to the growth period of the 
late 1950s and the 1960s. Recognition of the field as a separate specialty 
within higher education administration increased following the first insti- 
tutional research forum in Chicago in 1961. 

One of the driving forces behind the initial growth of the profession 
was the requirement that colleges and universities report various kinds of 
data for planning purposes. As institutions of higher education grew in 
size and complexity, both internal and external entities began requiring 
data by which to monitor their development. The fact that the growth 
was aided in large part by state and federal funds, even at institutions 
that did not receive direct subsidies from state coffers, added to the re- 
porting requirements, since governmental agencies had a responsibility to 
the taxpayers to show how the revenues were being spent. At the same 
time the development of the computer and analytical techniques, bor- 
rowed from operations research, made the summary and analysis of large 
amounts of data more feasible. 



External Reporting 

The types of surveys and other reports completed by institutional re- 
searchers for external entities are quite varied in terms of audience, tim- 
ing, and other characteristics. A number are periodic, being annual, semi- 
annual, each academic term, and so forth. Others are more sporadic, but 
still considered to be important enough to take the time and effort to 
complete them. While some reporting to external agencies is mandatory, 
as in the case of certain reporting requirements of state governing or 
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coordinating boards for state-supported universities, most is legally volun- 
tary. A number of reports, particularly to the federal government, may be 
classified as quasi-mandatory. These surveys cemnot be required by the 
U.S. government, but are considered to be so important by the govern- 
ment emd most colleges emd universities that they are considered to be 
memdatory for all practical purposes. Some of the most valuable reports 
from the institutional point of view, however, are the result of voluntary 
efforts among colleges emd universities, especially those in close geo- 
graphic proximity or with similar institutional missions. 

The HEGIS/IPEDS series of surveys are among the best known of the 
national, federally-sponsored reports. HEGIS is the acronym for the 
Higher Education General Information Surveys, a group of surveys con- 
ducted from the late 1960s until fall 1987, when they were succeeded by 
IPEDS or the Institutional Postsecondary Education Data Series. The 
HEGIS/IPEDS surveys, though var 3 dng in number emd subject each year, 
provide some of the best national data regarding higher education stu- 
dent enrollments, faculty numbers emd salaries, institutional finemcial 
measures, emd library holdings. The relative consistency of definitions 
over time has enabled this data source to be quite useful for historical 
comparisons. While definitional and other problems existed in the HEGIS 
surveys emd continue to exist in the IPEDS era, this series of reports 
remains one of the best sources of baseline data, particularly at the na- 
tional level. Most institutions of higher education have recognized this 
fact for years and have placed a high priority on responding to the surveys. 

Another well established emnual survey, this one dealing exclusively 
with faculty salaries, is conducted by the American Association of Univer- 
sity Professors (AAUP). For a time, the HEGIS survey of faculty salaries 
and the AAUP survey were both conducted each year, despite their simi- 
larity to each other. In recent years the HEGIS/IPEDS series has been 
more sporadic in nature, due to funding limitations, so the AAUP series 
provides the best data available nationally as to average faculty salaries 
by remk emd gender over a number of years. The AAUP definitions have 
also been more consistent them those of HEGIS emd IPEDS. One of the 
best features of the AAUP survey is the fact that the results are reported 
annually by institution in Academe, the AAUP publication, so they are 
widely available. The broad range of institutions, large and small, public 
and private, which submits faculty salary data to AAUP also encourages 
greater utilization of the results. 
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As noted above, the AAUP survey is conducted by a professional or- 
ganization of faculty, while the HEGIS/IPEDS data have been gathered 
by the federal government, more specifically by the Department of Educa- 
tion. Other federal agencies survey colleges and universities as well. 
Among the t)^es of surveys with which institutional researchers are most 
likely to be involved are those conducted by the Office of Civil Rights 
(OCR), such as the Workforce Profile, by the Office of Federal Contract 
Compliance (OFCCP), and by the National Science Foundation (NSF). 
The two former groups focus on possible illegal discrimination, while the 
latter organization gathers data on research expenditures and scientific 
manpower. The NSF data are published and widely distributed, so they too 
are quite useful in making comparisons across institutions and over time. 

Most states require a number of reports and sirrveys from state-sup- 
ported institutions. The range and variety of these are extensive and 
seemingly limited only by the creativity of the bureaucrats working for 
state governing or coordinating boards; data on student enrollments, 
numbers and types of faculty, space utilization, and expenditures are 
among the most common. Ideally the information is to be used for state- 
wide planning purposes, but in virtually every state there are reports 
which are done each year that are never used and whose purpose has long 
since been forgotten. The reason given for still requiring such data is 
usually that they may be useful for something at some time in the fu- 
ture— a response that is not normally well received at the campus level. 
Similar reasons have been known to be given by campus administrators 
regarding internal reports as well, however. 

When state level reporting is well done, it can provide information 
that is useful for both statewide and institutional planning purposes, 
while actually saving the colleges and universities time and money. Easily 
accessible data by institution, for example, eliminate the need for each 
institution of higher education to survey the others when making com- 
parisons. Aggregated to the state level, the data also provide a much 
broader perspective than is available at any one college or university. A 
number of states have been able to convince private institutions to partici- 
pate voluntarily in such data gathering efforts, in order to provide a more 
comprehensive statistical view of the status of higher education in the state. 

A recent development at the state level, particularly in the South, is 
the requirement that institutions report data on student learning, often 
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using the term outcomes assessment to describe these activities. (See 
Ewell [1985] for a summary of several models for assessing outcomes.) 
The reporting requirements, and instructional and evaluation activities 
which xinderlie the data, vary in their rigidity, but most institutional 
researchers in states that have mandated outcomes assessment reporting 
have been affected in some way. With the Southern Association of Col- 
leges and Schools (SACS) and a number of other regional and disciplinary 
accrediting bodies showing interest in outcomes assessment, it would 
seem reasonable to suggest that such efforts will continue to involve 
many institutional researchers. While the implementation of outcomes 
assessment requirements and reporting has been proposed as a tool to aid 
curriculum planning at colleges and universities, it is too early to predict 
how useful outcomes assessment will be to the planning process at the 
institutional level over the long term. 

In addition to the comprehensive national and state surveys, there are 
a number of other reports that are national or regioneil in scope and which 
offer useful planning information when the data are summarized. The 
Office of Institutional Research at Oklahoma State University (OSU), for 
example, annually surveys a number of larger xiniversities to obtain aver- 
age faculty salaries by academic discipline and faculty rank, including a 
category of new assistant professors. Faculty salaries by academic field 
are not available from another single source for such a broad range of 
disciplines. Even the smaller college can use the OSU data to gauge the 
‘‘market value” of new Ph.D.’s in departments where faculty vacancies 
occur. At larger institutions, the information can be used to compare, by 
department or other academic unit, how well the faculty members in each 
discipline are being compensated vis-a-vis the national norm. 

The College and University Personnel Association (CUPA) annually 
conducts an administrative salary survey analogous to the ones done for 
faculty salaries. The results are widely available and appear in abbrevi- 
ated form in The Chronicle of Higher Education. While CUPA does not 
have a category for newly appointed administrators, the data are reported 
by broad institutional category, based on student enrollment. Search com- 
mittees and administrators tr)dng to determine the average salary for a 
purchasing or bookstore manager, for example, find the CUPA data useful 
in establishing salary ranges. It is also a good resource regarding more 
traditional academic positions, such as college deans. Currently-employed 
administrators have been known to use the data in salary negotiations as 
well, both with those who report to them and those to whom they report. 
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Most colleges and universities belong to one or more voluntary organi- 
zations of similar institutions, such as the Association of American Uni- 
versities (AAU), Association of American Colleges (AAC), American Asso- 
ciation of State Colleges and Universities (AASCU), National Association 
of State Universities and Land Grant Colleges (NASULGC), and so forth. 
As a group, these are sometimes referred to as the “One Dupont Circle 
crowd,” after the building in Washington, D.C. where most of them are 
housed. Most such organizations collect data from their members regard- 
ing tuition levels and other student costs, fall enrollments, and a number 
of other variables. Such information can be quite useful to individual mem- 
bers in planning, since they are limited to a group of institutions that 
have already identified themselves as having somewhat similar missions. 

Less formal structures for data sharing have also been established by 
colleges and universities, usually among a smaller group of self-selected 
“peers.” The level of detail tends to be greater, but with fewer institutions 
involved, than the national surveys. Examples of these include: the SUG 
25, a group of large universities in the South; the Urban 13, a group of 
urban universities; and the TUFTS/EDUCOM Data Sharing Project, a 
group of 50 private liberal arts colleges and 20 private research luiiversi- 
ties. (As an aside, neither the SUG 25 nor the Urban 13 have the number 
of members reflected in their titles.) The distinguishing feature of these 
groupings is that they are voliuitary and self-selected. Each sets its own 
definitions for data elements and determines what kinds of data are col- 
lected. Summarization and reporting procedures have been adopted over 
time on a consensus basis. Since the institutions involved determine what 
is useful, the results of these efforts tend to be used more internally at the 
participating colleges and luiiversities. The levels of trust and cooperation 
are also sufficiently high to make response to ad hoc requests from group 
members more likely. 

In addition to the surveys noted above, a myriad of other requests for 
data reach institutions of higher education annually. These range from 
those done by for-profit organizations publishing guides to prospective 
undergraduate and graduate students to those done by doctoral students 
in conducting their dissertation research. One caveat is in order regarding 
opinion surveys directed to top administrators such as presidents and 
academic vice presidents: the busier the person, the less likely he or she is 
to respond to a survey personally. Experienced institutional researchers 
quickly learn that the survey sent directly to the president was, in fact. 
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completed by a graduate assistant. Hence conclusions drawn from such 
studies should be treated as tentative at best. 



Internal Reporting 

The discussion thus far has concerned external reporting, with the 
internal use of the results of such surveys being highly recommended, but 
not usually the primary reason for their completion. More important to 
institutional planning and management, however, is the judicious use of 
internal reports. While some standard reports distributed within a college 
or university community are the result of external reporting, most inter- 
nal reporting should be tailored to the unique mission, issues, priorities, 
and traditions of the individual institution. Both the data collected and 
the mode of presentation have to be sensitive to local needs in order to be 
useful to decision makers. 

One of the most common products of the internal reporting function is 
a factbook. A factbook is simply a collection of the most important infor- 
mation concerning an institution, ideally presented in an interesting and 
easily understood format. An important aspect of the factbook to consider 
is the fact that it is, despite its seemly objective character, a political 
document, since it reports data that are seen to be important internally. 
This point is made well in the chapter on factbooks by Jim Nichols and 
Rich Howard in the A Primer on Institutional Research (Muffo & 
McLaughlin, 1987). 

As an example, a common graphic in a factbook would be a map of the 
United States, with the number of students attending an institution from 
each state noted on the map. This type of graphic makes the point that 
the institution involved draws students from a wide geographic area, 
implying a national reputation and possibly even a national mission. One 
might avoid including such a map in a factbook, however, if the state 
legislature were considering a restriction on out-of-state students at the 
institution due to dissatisfaction with past admission policies. The data 
are nonpolitical, but the point that they make may not be. 

Besides providing a statistical description of a college or university, a 
factbook serves the function of being the “official” data source. It is not 
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uncommon in higher education organizations to have data disagreements 
between units; the facthook should serve as the arbiter of such disagree- 
ments. A number of footnotes may be required on certain pages, but it is 
possible with much patience and negotiation to get most people to agree 
to accept the institutional facthook as accurate, so that disagreements can 
focus on policy differences rather than whether or not a certain number is 
correct. Such acceptance also enables the institution to develop trend 
data, in order to study patterns over time. One requirement is that the 
data be “census data,” that is, obtained at a fixed point in time. A com- 
mon example is enrollment data, which are often counted on the tenth 
day of instruction during an academic period such as a semester or quarter. 

A caveat is in order regarding both the external and internal reporting 
roles. The institutional research function, which exists even at colleges 
and universities without formal institutional research offices, is some- 
times limited to reporting. In the most extreme cases, people are limited 
to answering surveys, which are considered a nuisance, and the data 
generated are ignored otherwise. In other cases internal reporting is also 
done, but the function of institutional research is limited largely to re- 
sponding to surveys and generating reports. The role that institutional 
research can play in providing information for internal planning and 
management can go far beyond these relatively mundane activities, since 
a deeper understanding of the forces affecting an institution internally 
and externally can only improve its operation. This role frequently in- 
volves special studies. 



Special Studies 

A major activity of a strong institutional research operation is that of 
conducting special studies. These can be described as responses to ad hoc 
requests or implied requests for data that are not routinely gathered and 
summarized. Some can be relatively simple and require only a short 
computer program; examples would include such activities as determining 
the number of applicants for admission with high school grade point aver- 
ages of B and SAT or ACT scores above the 90th percentile, the number of 
faculty over 60 years of age, the classroom vacancy rate between 12:00 
noon and 1:00 p.m., and so forth. The easiest responses to such requests 
are those that can use existing data sources and which are relatively 
straightforward. 
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More challenging, and often more interesting, are those studies done 
in response to more complex questions. A few examples of typical complex 
questions are: the relationship between high school academic perform- 
ance, standardized test scores, and college academic performance for a 
group of students; a reasonable cut-off point for scores on the TOEFL 
(Test of English as a Foreign Language) for foreign students; the perform- 
ance of the special support progrgim for students with academic deficien- 
cies; and so forth. 

The variety of such questions and studies to respond to them is end- 
less, but the skills necessary to address them properly are more sophisti- 
cated than those required to respond to surveys. Special studies require a 
thorough understanding of the institution involved and higher education 
generally, the latter being necessary in order for the local situation to be 
put into a broader perspective. Unfortunately, many colleges and univer- 
sities do not seem to place a high priority on getting sound answers to 
such questions from whatever office serves the institutional research 
function, for they hire relatively low paid people without much training or 
experience in higher education institutions to establish and maintain 
their institutional research operations. The consequence is an office that 
spends most of its efforts in reporting and, when a few special studies are 
attempted, tends to “reinvent the wheel” by discovering things that have 
been well known elsewhere for a long time. There is also less of an ability 
to perform one of the most difficult, but absolutely necessary, institutional 
research roles: helping decision makers formulate questions that can be 
answered from data which are available or accessible. 

While most special studies are considered to be one-time affairs, it is 
not uncommon for them to reoccur. If a relatively simple data question 
has been asked more than once, consideration should be given to making 
the special report a recurring, standard report. More complex special 
studies may likewise be required on a periodic basis, but may not have to 
be done each year. A prirne exeimple of a comprehensive special study 
done on a periodic, but not annual, basis is the institutional self-study 
completed as part of the accreditation process. Complex special studies on 
important topics (e.g., academic advising or economic impact of the 
institution on the community) can be done every three to five years, with 
the results being available for use in the self-study process when it occurs. 
Such an approach is very much in the spirit of recent developments in 
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regional accreditation standairds which require evidence of a continual self- 
study process, as opposed to one Herculean effort every five or ten years. 

Much of the emphasis on special studies has focused traditionally on 
products, for example, an academic department, the library, the print 
shop, the admissions office, the alumni office, and so on. What is fre- 
quently overlooked, however, is the study of processes. For instance, a 
great deal can be leaurned by studying the decision-making process for 
student affairs policies, or the process for updating data elements on the 
personnel database, or the faculty appeals process. Probably the best 
approach is to study both the product (e.g., the computer center) and the 
process (e.g., policy development regarding the purchase and utilization of 
computing resources), since the two are so interrelated. 



Other Institutional Research Activities 

There are several other types of activities in which institutional re- 
searchers are involved which are not always associated with traditional 
institutional research offices. One of these is the coordination and support 
of those performing the institutional reseaurch function within other units 
of the college or university. Other administrators, such as assistant and 
associate deans, vice presidents, admissions officers, and even some fac- 
ulty, participate in institutional research activities on a regulau: or irregu- 
lar basis, in support of the missions of their respective units. Rather than 
discouraging such direct data access and studies, a progressive institu- 
tional research officer seeks to encourage and support those involved. 

Another activity performed at a central level in support of decentral- 
ized institutional research is acting as a clearinghouse for the reception 
and distribution of data among the various parties at an institution. All 
or most surveys coming into the organization can be coordinated through 
a single office, even when that office does not complete the survey, in 
order to minimize duplication of effort and to allow for an initial screening 
as to whether or not a response is desirable. Likewise, many faculty and 
staff find it helpful to begin looking for institutional information in a 
single place, such as an institutional research office. 

The institutional reseaurch office should also serve as a center of exper- 
tise in the process of institutional research as well as the products, such 
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as data. The staff should he well read eind up-to-date regarding trends in 
higher education generally, so that those within the institution cem learn 
from the experiences of other colleges and universities. In addition, some 
attempt should he made to monitor emerging trends in the larger society 
outside of higher education so that the institution cem plan for societal 
changes which are likely to affect it directly emd indirectly. Other t 5 ^es of 
expertise cem reside in offices of institutional research. As progenitors and 
recipients of numerous surveys, staff in these offices tend to develop sur- 
vey research skills that can he quite useful to others in the academic 
community involved in survey research; locally available survey research 
expertise can save a great deal of time, energy, emd potential grief if 
consulted beforehand. 

A relatively recent development in higher education circles has been 
the use of institutional research to support internal academic and admin- 
istrative program evaluation. A common approach is for a college or uni- 
versity to evaluate each unit on a cycle, often five years, emd to use data 
provided by institutional research and other sources in the evaluation proc- 
ess. An emerging trend is for units not undergoing periodic evaluation to 
request one be done by a relatively objective unit outside its immediate 
environment; institutional research offices have provided this type of inter- 
nal management consulting role in a number of instemces quite success- 
fully. 

In brief, the primary pvupose of institutional research is to provide ac- 
curate, readily available information for plemning emd management pur- 
poses in a timely fashion. To do this properly, a mechemism should be de- 
veloped for the periodic evaluation of the institutional research function 
and unit or units charged with carrying out that function. As with emy 
evaluation, it should attempt to provide information to those involved 
that will help them improve their performance and resulting level of 
service to the institution of higher education employing them. 
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Accreditation . . has moved from a primary emphasis on process 
and resources to increased concentration on results and learning out- 
comes” (COP A, p. 5). This new emphasis embraces the scientific manage- 
ment concepts that resulted in the development of institutional research 
offices in the early 1960s. The Commission on Colleges of the Southern 
Association of Colleges and Schools (SACS) is the first regional accredit- 
ing body to mandate these management concepts as part of accreditation 
requirements. It appears certain, however, to become a major component 
in accreditation activities across the nation. 

The addition of outcomes assessment to the accreditation process 
has critical implications for the conduct of the self-study. The purpose of 
this paper is to discuss the self-study process as it pertains to the area of 
institutional assessment/evaluation. Traditional measures and activities 
of the self-study have not lost their importance or should not be ignored, 
but the strategy outlined is to complement and, in some cases, provide the 
foundation and structure for the self-study. 



Self-Study Design 

In the past, the self-study concentrated on the development of re- 
source measures or inputs (number of degrees awarded, research budgets, 
number of applications, etc.) and documentation of the many institutional 
processes. Often it was up to the visiting team to evaluate the adequacy 
of these resources and make recommendations useful to the institution. 

The Criteria for Accreditation (Commission on Colleges, 1987) re- 
quires that the institution evaluate the adequacy of its processes and 
resources in light of its stated purposes or mission. This implies continued 
assessment, or evaluation, of the institutional outcomes to identify weak- 
nesses or trends that run counter to the mission of the institution. The 
results of these evEiluations are then the catalyst for corrective action, and 
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the process begins again. In this mode, the institution is truly studying 
and evaluating itself. 

It is interesting to note that the accreditation process has moved 
from a formative evaluation process to a summative evaluation process. 
The implication of this is quite significant, for now the self-study process 
should not be a two-year institutional evaluation activity sandwiched 
between eight years of “doing good things.” It is critical that the institu- 
tion develop the capacity to continually review and evaluate the effective- 
ness of its progreims in light of its stated mission, and adapt when and 
where necessary to correct weaknesses. In this environment, when the 
formal self-study is completed and the report is written for the visiting 
teeim, there should be no surprises. 

Implied above is the notion that the design of the self-study should 
include the results of the institution’s ongoing evaluation processes. In 
fact, it could be argued that the self-study should, in large part, be a 
compilation of the results of periodic evaluation studies and documenta- 
tion of corrective actions taken as a result of evaluation activities since 
the last SACS visit. (Obviously, this implies that such activities have 
been taking place on the campus and that the results and subsequent 
actions have been documented.) 

The best of all possible worlds in the area of institutional assess- 
ment is described above. Assuming that the above situation does not 
exist, the following describes how an institution might gear up to conduct 
an ongoing assessment process and, as such, put in place the critical 
elements of the self-study. It would seem that the first step toward the 
development of an inclusive ongoing institutional assessment progreim 
should be foimd in the institutional research fimction. And, in fact, on 
page 11 of the Criteria, SACS speaks to its importance by requiring that 
the fimction be regularly evaluated. 

There are resources that should be available to an institutional re- 
search office or function. It is important to remember that institutional 
research is a staff fimction, typically designed to support the management 
of the institution (Muffo & McLaughlin, 1987). Its creation or presence 
is not, nor should be, considered the campus institutional assessment 
imit. While a critical factor in the assessment activities, the institutional re- 
search function should be there to support planning and decision-making 
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activities of administrators across the campus who have programmatic re- 
sponsibilities. 

Jim Montgomery illustrated a conceptual model which described 
the components of an assessment program (purpose-assessment-purpose). 
(See preceding paper in this publication.) In Table 1, this conceptual 
model is “operationalized” by comparing its components to that of a typi- 
cal social science research design. In general, institutional assessment (or 
a self-study) can be thought as a series of focused research/evaluation 
studies. 



Table 1. An “Operationalized” Assessment Program 



Research Design 



Self-Study Design 



Statement of the Problem 

Literature Review 

Development of Research 
Hypotheses 

Methodology: 

Instruments 

Design 

Sample 

Procedures 

Data Analysis 

Review of Results 

Recommendations 
Identified Weaknesses 

Implications for Further 
Research 



Define/Reaffirm Mission 

Area Reviews by Subcommittees 

Development/Review of Area 
Goals and Objectives 

Methodology: 

Instruments 

Design 

Sample 

Procedures 

Data Analysis 

Review of Results 

Recommendations to Strengthen 



Develop Strategies to Evaluate 
Impact of Corrective Actions 
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On the premise that the function of institutional research is pri- 
marily one of staff support, only two of the above components would 
ordinarily fall under its purview: Methodology and Data Analysis. Other 
components of the study should be the primary responsibility of adminis- 
trators and/or faculty who are directly involved with the program under 
study. As is reiterated several times by the Commission (1987), it is 
critical that senior administrators be visibly involved in the process. At 
the institutional level, the president/chancellor and governing board must 
define or reaffirm the mission of the institution. At the program level, vice 
presidents/chancellors, deans, and directors must provide direction in the 
establishment of the goals and objectives of their program. Likewise, they 
must be instrumental in the interpretation of evaluation results and the 
development of strategies to address weak areas identified in the evalu- 
ation. In addition, the measures to be used to evaluate the effectiveness of 
these strategies need to be identified through negotiation between those 
with responsibility for the program and the institutional research staff. 

It should be noted that evaluation statements are present in all 
sections of the Criteria. While Section III specifically addresses the re- 
sponsibility of the institutional research function, it should be realized 
that the institutional research staff should be involved in evaluation of all 
components of the institution. In many areas they can provide a coordi- 
nating function, making sure that viable processes for evaluation are 
being developed by the individual subcommittees. 

This is not unlike the role of the visiting team member who has re- 
sponsibility for institutional assessment. In many cases, he/she will have 
defined expectations for evidence of assessment activities to other team 
members as they execute their particular assignments. 



Data Concerns and Analysis 

In the past, it seems that accreditation was primarily concerned with 
data. Table after table of data pertinent to one program or another was 
developed in an effort to demonstrate its processes and the quality of 
input and output. The accreditation process was very much data driven. 
The new emphasis on quality and appropriateness of output shifts the 
process to one driven by information. (See Jones, 1982, for a short but 
very readable discussion of data and information.) The point here is that 
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the data collected in the self-study will have to be analyzed and turned 
into information instead of simply presented in a series of tables. The 
institutional research function must collect and analyze data for the pur- 
pose of creating information to be evaluated qualitatively in reference to 
stated goals and objectives by senior administrators and faculty. 

Summative evaluation studies in the final analysis are almost al- 
ways concerned with the evaluation of trends. The impact of strategies to 
correct weaknesses will rarely be seen immediately. The trick then is to 
develop standard definitions and processes for data collection. With insti- 
tutional data, the process is often referred to as the creation of “census” 
files. In essence, a census file is a snapshot of the operating (and as such 
dynamic) file on a specific data each year or semester. A snapshot of 
student data files may be taken on the last day to add a class at the 
beginning of each academic session. Personnel or facility data on the 
other hand may be frozen (another word for taking a snapshot) once a 
year, usually in the fall. External demands t 3 rpically dictate the times 
when census files are created. 

It is critical that the census files be checked for accuracy and rec- 
ognized by the senior administrators as the institution’s official source of 
data to be used for all external reports. Using the census files as the 
single source of data in factbooks and other documents developed to sup- 
port planning and decision making across the campus will insure a consis- 
tent source of base data across all internal and external agents. A second 
attribute of consistently developed census files is that trend analyses are 
not contaminated by calendar aberrations and questions challenging data 
compatibility or comparability from year to year. 

Trend analyses can also be conducted using data collected through 
surveys. Peter Ewell (1985) describes a series of surveys that when re- 
peated periodically over a number of years will provide trend information 
about student outcomes. 

A third important type of data in a self-study is that which allows 
the comparison of an institution to peer institutions. Again, comparisons 
with like institutions in a number of areas, over a period of several years, 
will result in trends that may identify intended, or unintended, gains or 
losses by an institution with respect to its peers. 
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The above discussion has centered around quantitative data that 
are usually manipulated through a computer. Another extremely impor- 
tant type of data in a self-study, however, is qualitative. This t 5 q>e of data 
is often found in the form of minutes taken during task force or committee 
meetings. Review of this type of data can reveal the type of concerns 
discussed and the data used in the development of policies or courses of 
action taken. It is possible that review and analysis of this t 5 q>e of data 
will result in insights of how decisions are made on the campus. 

The first and last three components of the self-study/evaluation 
design presented above are qualitative. Review of these decision making 
processes can reveal important insights about the extent with which senior 
administrators have been involved in the assessment of their institutions. 



Conclusions 

The new accreditation requirements of SACS demand that institu- 
tions continuously review and evaluate themselves in relation to their 
mission. Institutional assessment implies that all components of the insti- 
tutions review their effectiveness in meeting their goals and objectives. 
The purpose of this paper has been to provide some insight that might 
help in the development of an ongoing assessment process and, as such, 
prepare the institution for its self-study. These ideas are neither new nor 
novel. The institutional research community has been discussing and 
writing about them for the past 25 years! What is exciting, however, is 
that the scientific principles of management will become a visible part of 
the ongoing decision making and policy development process of evaluation 
of the institution’s effectiveness with respect to its goals and objectives. In 
the “Introduction” to Resource Manual on Institutional Effectiveness 
(1987), the Commission on Colleges list five institutional benefits of this 
type of management. In general, however, the primary benefit of institu- 
tional assessment is better-informed management of the campus. 

While, from an institutional standpoint, the benefits of an ongoing 
assessment program are potentially considerable and can help the institu- 
tion to successfully meet its mission, the key is the involvement and sup- 
port of senior administrators. From an operational or program standpoint. 
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the self-study process, while still time consuming and demanding, pro- 
vides a structure that should be less disruptive of the institution’s normal 
activities. If an ongoing evaluation process is in place, the self-study proc- 
ess and report should be (1) the coordination of past evaluation studies, 
(2) a review of the documentation of corrective actions taken in response 
to the ongoing evaluation process, and (3) evaluation of the effectiveness 
of the corrective actions taken. 

The primary thesis of this paper is that when an institution has in 
place a well thought-out ongoing assessment program, its self-study, by 
definition, is ongoing. The critical point is that for two years before the 
visit, the institution is not swallowed up by hectic data collection and 
point-in -time analyses. Instead, the self-study committee and its many 
subcommittees could be freed to assess the past direction and quality of 
the institution and its outcomes and to develop strategies to shape its 
future. 
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INSTITUTIONAL RESPONSE CAPABILITIES 
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This conference is addressing a topic that evokes great interest among 
institutions of higher education-especially if the institution will be under- 
going reaccreditation within the next few years. Even colleagues on cam- 
puses that will not be facing reaccreditation until the mid-1990s are ex- 
pressing concern about what efforts they should initiate now in order to 
demonstrate “institutional effectiveness.” The term, institutional effec- 
tiveness, may draw conference attendees in part because of the sense of 
panic at what seems to be an overwhelming task. 

So, there is concern about what it means to demonstrate institutional 
effectiveness. Yet what are institutions actually doing in preparation for 
accreditation under the new criteria established by the Southern Associa- 
tion of Colleges and Schools (SACS)? Are institutions at ground zero, with 
virtually no ongoing, campuswide activity related to the demonstration of 
institutional effectiveness? Or are some campuses well prepared with 
their staff mobilized to show that their institutions are accomplishing 
their purposes? Are resources available to support these efforts? Are some 
campuses better prepared than others? If so, what characterizes these 
campuses? 

These are the questions which Dr. Karen (jentemann. North Carolina 
State University, and I attempted to answer when we surveyed 311 insti- 
tutions that will be coming up for accreditation by SACS from 1987 to 
1991. The survey attempts to shed light on the capabilities of institutions 
to respond to the institutional effectiveness criteria. Specifically, the re- 
search addresses the extent to which institutions that will be reaccredited 
within the next five years by SACS are currently assessing institutional 
effectiveness. Differences in institutions’ involvement in assessment ac- 
tivities and in the support for the office responsible for assessment are 
compared for public versus private institutions, by the level of institution 
(community/technical colleges, four -year degree only, comprehensive insti- 
tutions, and doctoral-granting universities), and by enrollment (high vs. 
low). Of the 311 institutions surveyed, 167 institutions responded for a 
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response rate of 54 percent. The institutions participating are representa- 
tive of the sample when compared hy governance, level, enrollment, and 
state. 

A brief description of the procedures follows. The chief executive offi- 
cer received a letter with the questionnaire attached. The letter asked 
that he/she direct the questionnaire to “the person who will provide the 
data and analysis to support the evaluation process” required for the 
institution’s accreditation. The letter stated that in many cases the per- 
son will he the director of institutional research, hut that persons with 
other titles may he assigned the responsibility. 

The questionnaire contains 31 items describing evaluation, planning 
and institutional research activities required by SACS to demonstrate 
institutional effectiveness. For each activity, respondents indicated if it 
was “performed systematically and campuswide,” “either not systematic 
or . . . not campuswide,” “unknown,” or “not applicable.” For the purpose 
of this paper discussion will cover the percentages responding “performed 
systematically and campuswide,” as required by SACS. In addition to the 
31 items, 6 questions ask the availability of resources to the offices of the 
respondents. 

Responses were compared by governance, level, and size of the institu- 
tions. The relationship of two derived variables to responses were also 
examined. These variables are the function of the respondent’s office and 
the existence of an institutional research office. The function of the office 
was coded, based on the respondent’s title, with titles designating respon- 
sibilities for research, evaluation, and/or planning constituting one cate- 
gory and all other titles grouped into another category. The existence of 
an institutional research office was based on an examination of the organ- 
izational charts and the inclusion of the director of institutional research 
in the Higher Education Directory (1987). 

Results 

The majority of institutions that responded were under public control 
(65%). Approximately half (46%) were Level I institutions; 21 percent. 
Level II; 20 percent. Level III; and 13 percent. Level IV. Of the 108 public 
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institutions, the majority (60%) are Level I; in contrast, of the 59 private 
institutions, 51 percent are Level II institutions. 

Sixty-one percent of the 167 institutions do not have formal institu- 
tional research offices. A slightly higher proportion of private as compared 
to public institutions do not have institutional research offices (65% vs. 
57%). Only 41 percent of the respondents work in offices of research, evalu- 
ation, or planning. The range of respondents’ titles is broad, from chief 
executive officer to faculty serving as chair of the self-study committee. 

When the 31 items were treated as a single scale measuring “institu- 
tional effectiveness response capability,” differences that reached statisti- 
cal significance for two of the institutional characteristics were found. 
Overall, public institutions are engaging in more activities to demonstrate 
effectiveness than private institutions. Also two-year community, techni- 
cal and junior colleges are performing more of the activities than the 
baccalaureate-granting institutions. Size, the existence of an institutional 
research office, and the function of the office given responsibility for sup- 
porting the institutional effectiveness activities did not relate to scores on 
the scale. 

How did institutions respond to the specific items that are require- 
ments by SACS in order to meet the institutional effectiveness criteria? 
To answer this question, the items are grouped into content areas for ease 
of presentation. 

Planning Activities: Not quite three-fourths of the institutions have cur- 
rently established procedures for institutional planning and evaluation. 
In other words, over 25 percent have yet to even begin the process. About 
two-thirds have facilities plans. Two-thirds also have defined a process for 
curricular planning, review and evaluation. Slightly over half have a 
maintenance plan, a plan for the assignment of faculty responsibilities, 
and can demonstrate that educational planning guides budget prepara- 
tion. Slightly less than half carry out research studies of institutional 
purposes, policies, procedures, and programs-the heart of the assessment 
of institutional effectiveness. Only a third are able to demonstrate that 
institutional research supports planning— not surprising when only 39 
percent of the institutions even have institutional research (IR) offices. 
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More differences across types of institutions were found on the items 
relating to planning than to other content areas. Institutions with larger 
enrollments were more likely to have procedures established for institu- 
tional planning and evaluation, to have facilities plans, and to demonstrate 
that IR supports planning. If the respondent’s office had responsibility for 
research, evaluation, and/or planning, then the institution was also more 
likely to have established procedures for planning and evaluation and to 
demonstrate that IR supports planning. These institutions were also more 
likely to report carrying out studies of purposes, policies, procedures, and 
programs. Institutions with IR offices were more likely to report that they 
demonstrate that IR supports planning-not a surprising finding. Public 
institutions were more likely than private institutions to have facilities 
plans and maintenance plans. However, level of institution-community/ 
technical colleges, senior colleges, comprehensive universities, doctoral 
universitieS”did not relate to these items. 

Evaluations of Faculty, Instruction, and Administrators: The item which 
received the most positive responses was evaluations of full-time faculty, 
with 91 percent responding affirmatively. Somewhat unexpectedly, fewer 
of the Level IV, doctoral-granting institutions report carr 5 ring out cam- 
puswide faculty evaluations on a regular basis (71%) than institutions at 
the other levels (Level I, 94%; Level II, 91%; and Level III, 97%). Simi- 
larly the level of the institution was related to evaluations of part-time 
faculty. More community and technical colleges conduct regular campus- 
wide evaluations (86%). Sixty-six percent of Level II, 63 percent of Level 
III, and 50 percent of Level IV conducted systematic, institutionwide 
evaluations of part-time faculty. Almost three-quarters of the institutions 
have guidelines for the use of faculty evaluations, and two-thirds conduct 
studies of faculty workload. Over 50 percent study the effectiveness of 
instruction and also evaluate administrators. Less than 50 percent evalu- 
ate graduate teaching assistants (46%) and document that evaluations 
are used to improve teaching (43%). Although most institutions are evalu- 
ating faculty, the results are not necessarily being used to improve teach- 
ing. Interestingly, only the level of the institution is related to any of 
these items. 

Student Outcomes Assessment: The new focus of accreditation is on results, 
and student outcomes is the major emphasis. The picture, however, is not 
positive, in that only 41 percent are currently defining expected student 
outcomes, and only 44 percent are evaluating the outcomes. The 
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discrepancy in the percents implies that some institutions attempt to 
measure student outcomes before they have identified expected outcomes. 
A third of the institutions recommend methods of educational assessment, 
and only 29 percent can show that evaluation of students discriminates 
high and low achievement. No differences hy type of institution were 
found on these items. 

Public Service, Research, and Institutional Research: Areas that have re- 
ceived less attention than student outcomes hut may he just as important 
in the assessment of institutional effectiveness are evaluations of the 
public service mission, the research mission, and institutional research. 
Less than half of the institutions report systematic, campuswide activity 
in any of the six items relating to these content areas. Slightly less than 
half evaluate off-campus programs. About two-thirds of Level I (commu- 
nity and technical colleges) evaluate off-campus programs as contrasted 
to 33 percent of Level II, 33 percent of Level III, and 19 percent of Level 
IV institutions. Similarly, less than half of the institutions have goals for 
continuing education and extension, but no differences by tjrpe of institu- 
tion were found on this item. Only one-third of the respondents evaluate 
the institution’s research mission (responses limited to those for which the 
item was applicable). Similarly, a third evaluate institutional research; 
however, responses to this item differed by type of institution, with larger 
institutions (44%) more likely to do so than smaller institutions (23%). 
Also institutions with IR offices and those offices that had responsibility 
for research, evaluation, and/or planning were more likely to evaluate 
institutional research. Less than a third document the effectiveness of 
continuing education and extension, and less than a foimth evaluate the 
public service mission. More public (30%) than private (11%) institutions 
evaluate the public service mission. 

Resources: Responses to the questions relating to resoimces varied greatly 
depending on whether the question referred to the office’s present respon- 
sibilities or to all the activities required to assess institutional effectiveness. 
In general, the responses indicate that most consider their resources ade- 
quate to conduct current activities, but not for assuming all assessment 
efforts to demonstrate institutional effectiveness. The majority (55%) indi- 
cate that their budgets are adequate to carry out present responsibilities, 
whereas 77 percent state that their budgets are not large enough to carry 
out all the planning, evaluation, and research activities mandated by 
SACS. However, 55 percent state that their staffs are not large enough to 
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carry out even their current responsibilities; and 81 percent indicate that 
their staffs are inadequate for all activities pertaining to the evaluation of 
institutional effectiveness. Whereas two-thirds of the institutions state 
that their staffs have sufficient knowledge, skills, and experience for per- 
forming their current jobs, 55 percent say that their staffs do not have the 
expertise to assume full responsibility for the evaluation of institutional 
effectiveness. 

Responses to the six items on resources did not differ by governance, 
level, or student enrollment. The existence of an IR office related to this 
item: “My staff is large enough to carry out all activities . . . .” Institutions 
with institutional research offices were less likely to agree with this state- 
ment than those without IR offices (1.63 vs. 2.01). Respondents who were 
in offices responsible for research, evaluation, and/or planning were also 
less likely to agree with this statement than those whose offices had other 
functions (1.66 vs. 2.01). In other words, those with formal research of- 
fices and responsibilities for research showed less confidence that their 
staff was large enough to handle the new requirements. 



Discussion and Implications 

Overall, public institutions and community, technical, and junior col- 
leges report more activities in support of the demonstration of institu- 
tional effectiveness. However, the results from the survey did not identify 
major systematic differences across types of institutions in the assess- 
ment of institutional effectiveness. The few differences that emerged were 
to be expected. For instance, it was anticipated that larger schools would 
be more likely to have ongoing institutional research activities and that 
community colleges would be more likely to evaluate off-campus programs 
and part-time faculty. 

The disconcerting fact is that less than half of the institutions report 
evaluation and planning activities in 16 of the 31 areas. The overall 
picture is not positive. The majority of the institutions have not begun 
mobilizing to meet the new requirements in the areas of student outcomes 
assessment, the evaluation of the research and public service missions, 
and the evaluation of continuing education and extension programs. 
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The majority of institutions have not approached student outcomes as- 
sessment as an ongoing activity with an institution wide perspective. Yet 
all regional as well as specialized accrediting agencies are now emphasiz- 
ing outcomes. Only 44 percent have defined expected outcomes, the first 
step toward the development of assessment procedures. Even fewer insti- 
tutions have recommended or selected ways of evaluating the achieve- 
ment of educational outcomes. Within the next few years these institu- 
tions must demonstrate that they are not only evaluating the achieve- 
ment of student outcomes but that they are also using this information in 
program planning and improvement. Certainly the results indicate that all 
institutions, regardless of size, governance, or level, need to begin the first 
step in student outcomes assessment: the statement of expected outcomes. 

Despite the emphasis by SACS on evaluation of all institutional goals, 
few institutions are systematically addressing their research and public 
service missions. What are the expected outcomes of research and public 
service? What assessment procedures are appropriate in these areas? What 
criteria should be used to judge the effectiveness of research and public 
service activities? Clearly those institutions with broad missions of teach- 
ing, research, and public service will find that their efforts at evaluation 
must be even more comprehensive and will reqxiire more resources than 
institutions with the more specific mission of undergraduate education. 

Another area of concern is whether institutions have designated ad- 
ministrative responsibility for institutional research. The fact that only a 
third currently can show that institutional research supports planning 
and can evaluate the effectiveness of institutional research implies that 
responsibility may not yet have been assigned. In this sample only 39 
percent have formal institutional research offices, and only 42 percent of 
the offices responding to the survey even function as research, planning, 
and/or evaluation offices. Before accreditation, such responsibility should 
be assigned, the linkage to planning and evaluation must be evident, and 
the effectiveness of the activities must be assessed. It appears likely that 
new organizational units with institutional research responsibilities will 
emerge within the next few years. 

A disturbing finding to those of us who work in institutional research 
offices is that the existence of a formal institutional research office has 
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limited effect on current assessment activities. Perhaps Pace’s description 
of the dilemma of institutional research offices provides insight into the 
reasons: 

One might suppose that the institutional research office within the 
college would be the natural locus for ongoing institutional case 
study. But it might not be. Most such offices are beset by deadlines 
and heavily involved in basic accounting activities related to 
budget making, cost analysis, and similar matters, all of which 
orient the staff and its activities to serve administration and man- 
agement ... A case study needs data, in large amounts, but it also 
requires time for exploration, for reflection, and for thoughtful 
evaluation. Some institutional research offices have the capacity 
for educational evaluation as well as institutional accounting. 
Some do not. (Pace, 1979, p. 124) 

Not surprising in this study is that those serving in an institutional 
research capacity were most likely to indicate insufficient staff resources 
to carry out the full array of evaluation and planning activities. They, 
better than anyone, know the dilemma described by Pace and the extent 
of work required to conduct quality evaluations. These findings lead to 
recommendations for all institutions- whether public or private, doctoral 
or two-year, large or small. 



Recommendations 

First, all institutions shoiild designate an administrative unit respon- 
sible for institutional research. These offices must be clearly designated 
as having responsibility for assessment, and they must not be weighed 
down by “accounting” needs. Furthermore, the units must support institu- 
tional planning. Thus, the positioning of the unit within the organization 
is critical. 

Second, the research office must be sufficiently staffed so that it can 
carry out the “case study” of the institution. Across institutions, nearly 
half of the respondents need more staff just to carry out present activities. 
To assume responsibility for all assessment activities, the offices will need 
larger budgets, more staff, and more experienced, knowledgeable person- 
nel. As Pace suggested nearly a decade ago, the usual criteria used in 
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selecting an institutional researcher (i.e., a knowledge of computers and 
skill with data systems) are not necessarily the ones needed for quality 
evaluations. Rather, the desirable talents include . a historical and 
philosophical view of the institution and of higher education in general; a 
knowledge and an understanding of curriculum and teaching and of stu- 
dent development and learning; . . . and the ability to select and collect 
information about the institution’s effectiveness in fulfilling its diverse 
purposes” (Pace, 1979, p. 124). 

Si m ilar concerns were expressed by Van Maanen (1987) in his keynote 
address to the Association for Institutional Research, in which he called 
for institutional researchers to expand their horizons beyond the tabula- 
tion of data and into a contextual assessment of their institutions. Thus, 
the addition of knowledgeable staff with a broad view of institutional 
research is perhaps more critical than the mere expansion of staff. Experi- 
ence in program evaluation, test development, and assessment of student 
development is more critical now than in the past. 

Third, college and university administrators, governing boards, 
alumni, and legislators must recognize that costs are associated with the 
new accreditation requirements. In the area of student outcomes research 
alone, which the majority of institutions have not begun, the costs can 
easily run into thousands of dollars, as institutions either purchase exist- 
ing instruments or undertake the complex, timely, and expensive process of 
locally developing and validating instruments. Institutions cannot simply 
add on to current tasks without increasing their budgets for assessment. 

Most institutions have a long way to go before they can demonstrate 
institutional effectiveness. The breadth of activities has been described. 
But what these results do not tell us is the depth of the evaluation efforts. 
When an institution responds affirmatively that it carries out studies of 
institutional purposes, policies, and procedures-what does this mean? 
What kind of studies, and what is the quality of the “case study?” How 
useful are these studies in planning? The results of the survey do not 
allow assessment of the adequacy of the activities used to evaluate effec- 
tiveness. The conclusion, however, is that some institutions have yet to 
begin this task of stating goals and objectives, defining appropriate indi- 
cators of success, and developing assessment procedures that are a regu- 
lar activity— not a once-every-ten-years response to reaccreditation. 
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MINORITY RECRUITMENT, PERFORMANCE, 
AND RETENTION 

Michael T. Nettles 
Educational Testing Service 



In the United States there has been a great deal of progress over the 
past three decades toward increasing access for minorities into higher 
education. Thomas and Mingle has documented that dining the 1960s 
and early 70s, minority increases in higher education exceeded those of 
majority students if entering preparation, SAT scores, and so forth were 
taken into consideration. There has also heen a great change in the distri- 
hution of minority students hy type of institutions. Mingle noted that 
before 1950 nearly 100 percent of black students in the South attended 
predominately black institutions. Today, approximately 27 percent of 
undergraduate black students nationally and 50 percent in the South 
attend predominately black institutions with the other black students 
enrolling in predominately white institutions. 

Over the past ten years, however, there has been a decline in black 
student enrollment in undergraduate institutions. Between 1976 to 1984 
minority representation increased by 15 percent. In looking at individual 
racial/ethnic groups, however, Asian and Pacific islanders constituted the 
greatest amount of that growth, with a 63 percent increase in Asians. The 
black undergraduate enrollment over the same period declined by 8 per- 
cent and black graduate enrollment declined by 21.6 percent. 

This black enrollment decline coincides with an era in which financial 
aid changed. Much of the growth in black and Hispanic enrollment during 
the 60s and 70s is attributed to increases in financial aid. In 1980 about 
17 percent of all financial aid was in the form of loans with the balance 
being in the form of grants and scholarships. Today the distribution is 
fifty/fifty. This distribution of financial aid certainly makes a difference in 
desire for and access to higher education, and especially affects under- 
graduate and graduate enrollment of black students. 

Of all segments of higher education a greater percentage of minority 
students are at community colleges than at other types of institutions. 
Blacks constitute 8 percent of undergraduate enrollment in four-year 
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institutions and 10 percent in community colleges. A desirable goal for 
society would appear to be for black and Hispanic minority groups to 
represent at least the proportions of undergraduate and graduate enroll- 
ments that they represent in the general population. Blacks represent 
slightly over 12 percent and Hispanics represent nearly 7 percent of the 
population, but the further along these two minority groups move in the 
pipeline, the smaller their representation becomes. In terms of retention, 
although blacks represent 8 percent of undergraduate enrollments, they 
represent only 5 or 6 percent of the baccalaureate degree recipients each 
year. This demonstrates that blacks have a higher attrition rate. 

The problems of outcomes and institutional effectiveness have to and 
should take minority issues into account. In fact, it is one of the most 
compelling reasons to conduct institutional effectiveness studies. As an 
example, although blacks represent 12 percent or better of the population, 
only 2.4 percent of professional engineers, 5.4 percent of mathematicians 
and computer scientists, 3.2 percent of natural scientists, 5 percent of 
physicians, and .9 percent of dentists are black. In all professions, minori- 
ties are under-represented. 

The American educational system as a whole has been slow to im- 
prove the quality of education K through 12 or higher education. Grades 
K through 12 are indicators for near-future projections for higher educa- 
tion. Large numbers of black children still attend schools that are pre- 
dominantly black. The uncertainty about black distribution at the K 
through 12 level and the quality of education being provided to students 
at those levels still exists. Annually the differences in the performances of 
these students in elementary and secondary schools on measures of the 
National Assessment for Education Progress are documented, and yet 
very little progress is made in eliminating the gaps. Even at the point of 
admission to college, blacks perform 100 points, on average, below whites 
on the verbal section of the SAT and 114 points lower in the quantitative 
section of the SAT. 

This raises the question then of what does this represent for higher 
education institutions? It seems that many institutions do not know 
enough about their students, particularly their preparation and back- 
grounds, to serve them effectively. Educators often look at average or 
mean scores on the SAT, but know very little about the distribution of 
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scores or the t5T)es of high school curricula and remedial/developmental 
needs of students who took the test. More minority students in actual 
number are performing on the upper end of the scale now than two 
decades ago, hut educators know very little about the backgrounds and 
experiences of those people at the upper end compared to those minorities 
who are performing less well on the examination. Institutional research- 
ers must address this issue as they proceed with outcome or institutional 
effectiveness measurement. 

In conjunction with the Ford Foundation, I am currently working 
along with colleagues at Educational Testing Service (ETS) with five ur- 
ban community colleges to assist them in determining who is enrolled in 
their institutions, how well their students progress through the institu- 
tions, and what happens to their students after they graduate or complete 
the community college curriculum. Initially the lack of information that 
was available was astounding and national sources were sought to help in 
understanding the status of the enrollments and the characteristics of 
students enrolled in community colleges, but not very much useful infor- 
mation was available. 

The five community colleges, for example, did not have institutional 
research offices prior to beginning this project. The first step was to assist 
the faculty and staff in designing their own research projects. The deci- 
sion was made to track groups of students, randomly selected over the 
course of four years, to determine the stop-out, drop-out, retention rates, 
and ultimately transfer rates and/or occupational attainment. One initial 
goal was to select small samples so that the institutions could follow the 
subjects effectively. Although tracking the whole population would be 
desirable, maintaining records of 1,300 to 1,400 students for that period 
of time was impossible given the limited resources and person work-hours 
required. The institutional researchers were encouraged to consider mi- 
norities as one major stratification. When conducting institutional effec- 
tiveness studies, the results will be very revealing about the relative ef- 
fectiveness of minority students compared to majority students. 

Previous research conducted at over 30 institutions has revealed that 
black students performed on average a whole letter grade lower than ma- 
jority students. The factors that explain black students’ performance in 
college often are comparable to those for majority students. 
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There were several factors found to be significant predictors of grades 
for minority students: 

1. Feelings of racial discrimination were very important in explaining 
grades of both minority and majority college students. The degree 
of feelings of racial discrimination differed depending on the type 
of institution; however, black students reported higher degrees of 
feeling of racial discrimination than did majority students. White 
students on predominately black college campuses felt greater ra- 
cial discrimination than did black students on black campuses, but 
the effect of these feelings on white students’ grades was less than 
on black students at predominately white colleges. 

2. The number of interfering problems was also significant. Students 
with few personal problems perform better in college. 

3. High grade point averages in high school and high SAT scores 
predicted higher college grades for both black and white students. 

4. Satisfaction with the institution contributed to successful academic 
careers. Blacks on predominately white campuses reported lower 
satisfaction with the environment. 

5. Good study habits were a predictor of good grades for minority 
students. However, it was found that blacks have poorer study 
habits than whites. 

6. High academic integration, contact between students and faculty 
on matters perteiining both to course work and career develop- 
ment, was an influential factor. Students at institutions where 
faculty had a low level of influence on student development tended 
to perform higher in terms of grades, but this was not true of 
retention. Retention rates were found to be motivated by faculty 
involvement. 

7. Married students and students with relative strong peer relation- 
ships on campus were more successful in college. There was little 
difference in the peer relationships of black and white students, 
yet peer relationships were segregated racially. 
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8. Older students and non-transfer students tended to have higher 
grade point averages. One interpretation of this is that adjustment 
and continuity within the same institution helps encourage per- 
formance. 

9. Socioeconomic status, which is perceived as a predictor of perform- 
ance in college, was not significant in explaining student grades, 
but it was significant in retention and attrition rates. Forty-five 
percent of black students who attended these institutions were 
likely to graduate from those institutions over a period of five or 
six years as compared to 65 percent of the white students. Pro- 
grams of study completion rates increased at private institutions. 

10. SAT scores tended to over-predict the college performance of black 
students. Black students tended to perform lower than predicted 
by their SAT scores. This should have implications for policies on 
admissions as well as special programs at colleges and universities 
that are designed to contribute to minority student performance. 
Personal experience reveals that institutions tend to establish ad- 
mission policies using very broad policy consideration. For ex- 
ample, an institution of 25,000 students attempted to improve the 
quality of its student body. One approach implemented was to 
increase both retention standards and admission requirements, 
with no distinction made in the admission requirements for minor- 
ity and majority students. This resulted in a decline of enrollment, 
but a greater decline in minority enrollment in what previously 
had been an open admissions institution. The minority enrollment 
declined and still the minority students who were admitted subse- 
quently performed lower than the majority students admitted by 
the same standards. 

An examination is required for admission in Georgia and students 
falling below a certain level are placed in non-degree credit courses in 
developmental remedial programs. As a result, a higher percentage of 
minority students who are admitted to the institutions are enrolled in 
these developmental education programs. Institutional effectiveness stud- 
ies must examine the progress an institution makes in preparing these 
students for entering the mainstream of the institutional curriculum. 
Both Georgia and Florida administer general education outcome measures 
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and/or basic skills measurements at the junior level, resulting in some at- 
trition that is also greater for minorities than majority students. The 
questions to be reused regarding adequate minority representation in the 
institutions are; Why are minorities performing at a lower level on these 
measures after two years of college and what can be done to eliminate the 
differences? 

These are important questions. The level of adequate representation 
of minorities in graduate and professional schools, among graduate de- 
gree recipients, among college faculty, and in the professional and scien- 
tific work-force will never be reached until the quality of education for 
minorities in undergraduate institutions is improved. A focus of institu- 
tional effectiveness research must place emphasis upon minority issues, 
especially improving the quality of learning and development of students 
at undergraduate institutions. 
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The topic of minority recruitment, performance, and retention can be 
related to some of the work that is being done in terms of assessment and 
placement standards in higher education nationally and in the Southeast 
specifically. Harold Hodgkinson (1985) and numerous other researchers, 
and higher education organizations and commissions are engaged in 
demographic projections concerning the futiire makeup of the college-age 
popiilation. Without exception, these reports are projecting a futiire colle- 
giate popiilation that will be decidedly more “colorfiil” and diverse in 
appearance. 

Couple these projections with the findings of the National Assessment 
of Educational Progress-that minority students lag far behind their white 
counterparts in reading amd mathematics proficiency levels-and it is rea- 
sonable to suggest that large numbers of minority students will make up 
the college-age popiilation that is underprepared for college-level work. 
Perhaps a more important question is; how will institutions respond to 
these demographic amd preparatory changes in the college-age student 
pool? Whatever the response of higher education, there are likely to be 
profound social amd economic impacts— either positive or negative. The 
following example emphasizes the importance of academic preparation 
and a chamging workforce: today there are seven working people for every 
retired worker and by the year 2005 this will be reduced to three working 
people for every retired worker. This example makes it very clear how 
importamt a skilled and competent labor force will be in maintaining 
existing, socioeconomic standards and democratic ideals into the future. 

In times of declining enrollment there are two ways to increase or 
maintain enrollment. One way is to increase the diversity (ethnicity, SES, 
age cohort, etc.) of the enrollment pool and, secondly, retain a higher 
proportion of those students already enrolled. As the traditional college 
enrollment pool declines in the near futiire, circumstances will require 
colleges amd universities that wish to remain viable educational institu- 
tions to spend an inordinate amount of time and energy tr3ring to retain 
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more students, and especially minority students. As the competition in- 
creases for the remaining pool of students, it is very important for colleges 
and imiversities to pay closer, more focused attention to those programs 
that already exist and that are currently in place. Particular attention 
must he given to those programs that impact or play an important role in 
minority access, performance, and retention. 

Remedial/developmental programs seem like a logical place to begin to 
address these issues and needs at the college level. For one reason, almost 
every public higher education institution has at the very least an informal 
remedial program. Nationally, and in the Southeast region in particular, 
well over 80 percent of the public institutions have special support serv- 
ices and curricula for underprepared freshmen. Secondly, these programs 
have demonstrated the ability to increase retention rates for the margin- 
ally prepared student. For these reasons, and when numbers from around 
the coimtry indicate that up to 50 percent of all incoming freshmen-and 
even higher for minorities-are in need of additional academic prepara- 
tion, it is clear what an important and essential role these programs can 
play in raising student retention rates in general and minority rates in 
particular. 

Two examples where this has occurred are New Jersey and Tennes- 
see. In both of these cases there are state-mandated assessment and 
placement programs at the college level. Also, in each state’s assessment 
and placement program there are built-in information and data gathering 
procedures that document what is taking place in each program-assess- 
ing input and output characteristics of the curriculum, students, and fac- 
ulty. This issue is raised because the inclusion of this informa tion/data 
function as an integral part of the program design and structure is critical 
to the overall success that each program has demonstrated. Outcome 
assessments reveal that students entering colleges through remedial/de- 
velopmental programs have equitable, and in many cases higher, reten- 
tion rates than students who do not enter college through these programs. 
If in fact in the future more minority students will be entering college in 
need of additional academic support, it seems logical that through reme- 
dial/developmental programs colleges and imiversities will have an oppor- 
tunity to address issues of minority access, quality education, and in- 
creased student retention-imagine, one college-level program that has 
strong possibilities of addressing three important higher education issues. 
In both New Jersey and Tennessee, published results indicate retention 
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rates--to the second year and on to graduation--that differ only slightly 
between the two groups of students. 

Increasing minority retention in the future hy all indications will de- 
pend heavily on how the states and higher education institutions/systems 
develop policies to deal with remedial/developmental programs at the 
higher education level. Specifically, will colleges and universities he ahle 
to fully embrace these programs as viable and integral components of the 
institution’s normal mission and function, and will the states support 
them accordingly? If the answer is yes, then it is plausible for higher 
education to go a long way toward enhancement of minority recruitment, 
performance, and retention. 

One problem in higher education that keeps occurring and that inter- 
feres with information and knowledge collecting involves issues/questions 
that do not lend themselves to clear or simplistic answers. By the same 
logic, there are problems and issues in higher education that are solvable 
but only lack asking the right questions. In the case of the remedial/ 
developmental issue in higher education, it is not the case that issues and 
questions of student underpreparation or college-level placement cannot 
be answered or addressed, it is just that educators, administrators, and 
policymakers have not identified this as an important area of inquiry and 
as a result are not asking the right questions. The New Jersey assess- 
ment program was cited earlier as a good example of a system that is 
graduating greater numbers of minority students, and this as a direct 
result of their college-level placement program. What are they doing as 
compared to other states and institutions to achieve these results? Basic 
lines of inquiry such as this need to take place. In order to effect change 
and expand the knowledge base, educators, administrators, and poli- 
cymakers must realize the importance of fully understanding the value 
and impact remedial/developmental programs can have on increasing mi- 
nority participation in higher education. 
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KEY INGREDIENTS OF INSTITUTIONAL 
EFFECTIVENESS FOR MINORITIES 

Joseph Marks 
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Dr. Nettles’ central themes amount to an answer to the question, 
“What does the topic of minority retention, performance, and recruitment 
have to do with a conference on institutional effectiveness?” I would like 
to reinforce this tie between the conference theme and the topic of this 
panel, but will do so in slightly different yet quite sympathetic terms. 

The Challenge of Access with Quality: The economic and demographic 
shifts of an aging population and increased minority representation with 
regard to improving the future quality in higher education present chal- 
lenges or opportiuiities for progress. Such progress must be measured in 
terms of participation and access as well as in terms of traditional aca- 
demic standards. Policies which merely exclude students as a way of 
improving measured levels of achievement do not promote the kind of 
educational progress that we are going to need if our society is to meet the 
challenges of keeping its place in the international community, of keeping 
the strength of its democratic institutions, and of maintaining the life 
styles to which the American population has become accustomed. 

The challenge before institutions is to find ways that will enable the 
achievement of two goals simultaneously: access (increased participation) 
and quality improvement in the traditional academic sense. To meet both 
of those goals for the same students at the same time, colleges simply 
must do a bigger and better job with larger numbers of students for which 
they have historically done the poorest job. 

The Trends in the Enrollment of Black Students: The trends in the college 
and university enrollments of black students are by now widely reported 
and recognized (Marks, 1986, 1985, 1984, 1982; Mingle, 1987; ACE, 
1987). Unless current trends are reversed, there will be fewer black stu- 
dents in higher education in 1990 than there are today. Analysis of the 
trends also shows that even before the “quality improvement” rhetoric 
took center stage, the participation rates of black students in higher edu- 
cation were declining. 
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Since some quality improvement strategies can negatively affect the 
achievement of the access goal, it is essential to examine factors which 
influence the enrollment trends of black students in higher education. 

Demographic Influences on Enrollment Trends: It is important to increase 
the college participation of minority students, particularly of black stu- 
dents, in the futime because they are going to be a larger percent of the 
total society and the work force in particular. While minorities will be a 
larger percent of the population seeking higher education, niunerically 
there will be fewer in the college-age pool for the next ten years. With 
proportionally fewer black students seeking higher education and in- 
creased dropout rates of black students in high school, there is a further 
shrinking of the cohort of college-eligible black students. While the high 
school graduation rate of minorities has been increasing, it has not made 
up for the effects of declines in the size of the college-age cohort and of the 
declining proportion of black high school graduates seeking higher educa- 
tion. The only way to increase the representation of minorities from a 
demographic point of view is to start widespread and intensive programs 
to increase the high school enrollment rates of black students, to maintain 
the high graduation rates of those that remain in high school, and to 
increase the college participation rates of those graduating from high 
school (Marks, 1985). 

The Academic Influences on Enrollment Trends: The differences in academic 
preparation and performance of black and white students took genera- 
tions to create, and may take generations to solve. However, society may 
not be able to wait generations for the normal social and cultural develop- 
ment cycle. Differences of academic preparation must be attacked with a 
special vigor. Standardized test differences of a hundred points remain be- 
tween black and white students on some tests. In the recent pilot pro- 
gram that the Southern Regional Education Board (SREB) sponsored 
with the National Assessment for Educational Progress, state-by-state 
comparisons of high school student achievement levels in reading, writing 
and other subjects are striking. The average reading level of eleventh- 
grade black students is equivalent to that of seventh-grade white stu- 
dents (SREB, 1985, 1986, 1987). In writing, the results showed that only 
about 30 percent of the white students have writing skills that are judged 
to be adequate for college-level work, and far fewer black students (about 
15%) have such writing skills. Educational disadvantages due to social 
and economic disadvantages over the years are very persistent. 
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Evidence from a wide range of soiirces supports the proposition that 
academic preparation (a college preparatory curriculum) and academic 
performance are among the most important factors in promoting college 
attendance (Ramist, 1981). Extraordinary efforts are required to continue 
the elimination of educational disadvantages that are roadblocks to fiir- 
ther progress toward the access with quality goal. 

Student Financial Aid Influences on Enrollment Trends: Student financial 
aid on a per recipient inflation-adjusted basis has diminished substan- 
tially in the last eight years. Its distribution system is very complex and 
leads to the circumstance of some financial aid recipients receiving more 
money than their financial aid analysis reveals they need, while others 
receive far less than needed. This has a clear impact on economically 
disadvantaged students who are over represented among minority 
groups. 

Changes in the level of funds available for student financial aid and 
changes to make the distribution system more equitable, while helpful 
and necessary, are not the key ingredient in promoting access with qual- 
ity. Until preparation levels are improved substantially, significant 
growth in the numbers of black college students will not occiir. As one wit 
put it, without scholarship, scholarships do not help. Additional student 
aid funds can only make marginal differences so long as academic disad- 
vantages persist. 

Institutional effectiveness evaluation efforts must incorporate atten- 
tion to how the institution is serving minority students, black students in 
particular, because neither academic quality without access nor access 
without academic quality is effective higher education for the 1990s. 
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MINORITY PERFORMANCE 
IN ACADEMIC SETTINGS 

Nathaniel Pugh 
Georgia Southern College 



Dr. Nettles has published research results which identify factors that 
make a difference in student performance within academic settings. In 
addition, he has stressed the need to give attention to noncognitive meas- 
ures as well as continuing the use of cognitive indicators in making ad- 
mission decisions. As an admissions director at Stanford, I had the oppor- 
timity to admit students based not only on their admission scores or 
grade point averages, but on how I felt they could contribute to the 
academic atmosphere. 

Dr. Nettles remarked that colleges should not limit admission to just 
those black students who score 100 points less on the SAT than white 
students. He also said that institutional researchers should begin to ag- 
gregate data and look at its impact upon an institution’s admission poli- 
cies. Based on a research study in progress, fimded by the Southern 
Education Foimdation, I have collected aggregate data on academic and 
testing standards in five states. As a part of this research study, I used 
Georgia Southern College as an institutional pilot because the University 
System of Georgia in the fall of 1988 will require a new set of standards 
for college preparedness. 

The sample for the institution pilot used a random 10 percent of all 
entering 1984 freshmen and all entering 1984 black students. To date 
those students have been tracked for three years. In terms of college 
preparedness, both the random sample and the black sample demon- 
strated virtually no difference in the precollege high school curriculum. In 
terms of their high school grade point average, there was a slight differ- 
ence, 2.85 for the black students versus 2.70 for the other group. How- 
ever, there was a significant difference in the SAT scores. The 10 percent 
sample had an approximate SAT composite of 834 points, the black 
sample had 729, a score disparity of 100 plus points. What happened in 
three years in terms of performance and retention? What are the differ- 
ences, if any, in retention between these two groups? At the end of one 
year 65 percent of all freshman students remained in the college and 83 
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percent of all the black freshman students remained. By the end of the 
second year 49.6 percent of all the students remained while 67 percent of 
all the black students remained. Diiring the spring quarter of 1987, 41 
percent of the total sample of all students remained, and 55 percent of the 
black students remained. 

What are the differences, if any, in performance over this period of 
time between these two groups? The 10 percent sample of aill students 
had a GPA of 2.32, and the GPA for the black students was 2.23, almost 
no discernible statistical significance in the difference in the GPA. How- 
ever, the students that remained at the end of this three-year period had 
a difference of 68 SAT points between the two samples. Dr. Nettles is 
correct in stating that noncognitive areas may explain why students, 
black or white, persist and that SAT scores may have limited predictive 
power for academic performance of black students. 

According to a policy study by the Educational Testing Service, one of 
the best success indicators for preparing students to enroll in curriculums 
such as engineering or the health sciences is a summer science enrich- 
ment program. Georgia Southern has such a program and selects stu- 
dents with various SAT scores and a variety of noncognitive measures. 

It has been shown in the health sciences that there is a positive corre- 
lated relationship between the SAT and the MCAT. Our students are 
selected basically because they are either academically or economically 
disadvantaged. By the time these students are at the point of pursuing 
professional education, we have data that can support their opportunity 
to enroll in professional schools. The bank of noncognitive information 
includes 16 personality factors, the Myer Briggs, and Sedlacek’s instru- 
ment of noncognitive variables. We also used the Strong Campbell Inter- 
est Inventory for a limited number of students. This inventory revealed 
that the choices of students who selected medicine, dentistry, and phar- 
macy as careers were not related to their assessed interest patterns. We 
are finding that some students with excellent academic records and high 
SAT scores want to leave science and mathematics when they have com- 
pleted the summer science enrichment program. Institutions need to con- 
duct research studies regarding the relationship between noncognitive 
variables and cognitive variables and the predictive power of these vari- 
ables upon student performance. 
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Most of the instruments, whether cognitive or noncognitive, do not 
take into account the minority student population. These measures were 
not normalized on this population. There are numerous ways to get 
through college in terms of surviving the experience. These ways may not 
be measured by the variety of instruments in use today by academic 
decision makers. 



References 

Pugh, N., Jr. (1986). Assessing the new standards for high school gradu- 
ation and college admissions. In C. Fincher, L. Jones, & J. Placek 
(Eds.), Assessment, Evaluation, and Accreditation. Proceedings of the 
Third Annual Research Conference on Higher Education in Georgia. 
Athens: Institute of Higher Education, University of Georgia. 




115 



ABOUT THE AUTHORS 



Dr. Ellen Earle Chaffee is Associate Commissioner for Academic 
Affairs of the North Dakota Board of Higher Education and senior author 
of Collegiate Culture and Leadership Strategies (ACE/Macmillan, 1988). 

Dr. Alton L. Taylor is Professor of Higher Education and Director of 
the Summer Session at the University of Virginia. 

Dr. Ernest T. Pascarella is Professor of Educational Psychology in the 
College of Education at the University of Illinois at Chicago and co-author 
(with Patrick Terenzini) of the forth-coming book, Does College Make a 
Difference? 

Dr. Patrick T. Terenzini is Professor of Higher Education in the Insti- 
tute of Higher Education at the University of (Georgia and author of 
numerous studies dealing with the influence of college on students. 

Dr. James A. Montgomery is Director of Institutional Research at Vir- 
ginia Tech and a former president of the Association for Institutional 
Research. 

Dr. John A. Muffo is Assistant Director of Institutional Research at 
Virginia Tech and co-editor of A Primer on Institutional Research (AIR, 
1987). 

Dr. Richard D. Howard is Director of Institutional Research at North 
Carohna State University and co-author of two chapters in A Primer on 
Institutional Research. 

Dr. Brenda H. Rogers is Associate Director of the Office of Research 
Evaluation and Planning at North Carolina Central University. Previ- 
ously she was on the institutional research staff at North Carolina State 
University. 

Dr. Michael Nettles is Vice President for Assessment at the University 
of Tennessee. Prior to his appointment, he was Research Scientist at 
Educational Testing Service in Princeton, New Jersey. 



ERIC 




Dr. Ansley Abraham is Staff Research Associate for the Southern Re- 
gional Education BoEird and a frequent participant in Institute confer- 
ences and seminars. 

Dr. Joseph Marks is Staff Research Associate for the Southern Re- 
gional Education BoEird and a former staff member of the Georgia Gover- 
nor’s Committee on Postsecondary Education. 

Dr. Nathaniel Pugh is Director of the Florida Institute of Education in 
Jacksonville. Previously he was Special Assistant to the President of 
Georgia Southern College. 

Dr. Cameron Fincher (Editor) is Regents Professor of Psychology and 
Higher Education at the University of Georgia and Director of the Insti- 
tute of Higher Education. 



117 

ERIC 



PURPOSE AND FUNCTIONS 



The Institute of Higher Education is a service, instructional, and re- 
search agency of the University of Georgia. Established in 1964, the Insti- 
tute cooperates with other agencies and institutions in the development of 
higher education. Programs and services help prepare professionally 
trained personnel in higher education, assist other colleges and universi- 
ties in numerous ways, and study the organizational and functional pro- 
cesses of institutions and programs of higher education. 



The Institute publishes a series of occasional papers, monographs, and 
newsletters dealing with selected topics in higher education. The general 
purpose of Institute publications is to inform administrators and faculty 
members of recent trends and developments in areas such as administra- 
tion, cvuriculum planning, program evaluation, professional development, 
and teaching effectiveness. The specific intent may be to report research 
findings, to interpret general trends or recent events, or to suggest new 
lines of inquiry into various problems. 

Additional copies of this publication may be purchased for $6.00 from: 



Institute of Higher Education 
Candler Hall 
University of Georgia 
Athens, GA 30602 
404/542-3464 



Publications 




ERIC 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 



Reproduction Basis 




This document is covered by a signed "Reproduction Release 
(Blanket)" form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a "Specific Document" Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release form 
(either "Specific Document" or "Blanket"). 




EFF-089 (3/2000) 




